[jira] [Created] (HIVE-12058) Change hive script to record errors when calling hbase fails

2015-10-07 Thread Yongzhi Chen (JIRA) 
Yongzhi Chen created HIVE-12058:
---

 Summary: Change hive script to record errors when calling hbase 
fails
 Key: HIVE-12058
 URL: https://issues.apache.org/jira/browse/HIVE-12058
 Project: Hive
  Issue Type: Bug
  Components: Hive, HiveServer2
Affects Versions: 1.1.0, 0.14.0, 2.0.0
Reporter: Yongzhi Chen
Assignee: Yongzhi Chen


By default hive will try to find out which jars need to be added to the 
classpath in order to run MR jobs against an HBase cluster, however if hbase 
can't be found or if hbase mapredcp fails, the hive script  will fail silently 
and ignore some of the jars to be included into the. That makes very difficult 
to analyze the real problem.
Hive script should record the error not just simply redirect two hbase failures:
HBASE_BIN=$
{HBASE_BIN:-"$(which hbase 2>/dev/null)"}
$HBASE_BIN mapredcp 2>/dev/null



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-12059) Clean up reference to deprecated constants in AvroSerdeUtils

2015-10-07 Thread Aaron Dossett (JIRA) 
Aaron Dossett created HIVE-12059:


 Summary: Clean up reference to deprecated constants in 
AvroSerdeUtils
 Key: HIVE-12059
 URL: https://issues.apache.org/jira/browse/HIVE-12059
 Project: Hive
  Issue Type: Improvement
  Components: Serializers/Deserializers
Reporter: Aaron Dossett
Priority: Minor


AvroSerdeUtils contains several deprecated String constants that are used by 
other Hive modules.  Those should be cleaned up.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 37778: HIVE-11634

2015-10-07 Thread John Pullokkaran 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37778/#review101810
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/pcr/PcrExprProcFactory.java 
(line 384)


WALKERSTATE should be UNKNOWN.


- John Pullokkaran


On Oct. 7, 2015, 7:47 p.m., Hari Sankar Sivarama Subramaniyan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/37778/
> ---
> 
> (Updated Oct. 7, 2015, 7:47 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, Jesús Camacho Rodríguez, and John 
> Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Support partition pruning for IN(STRUCT(partcol, nonpartcol..)...)
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 54a529e 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 439f616 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/PartitionColumnsSeparator.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/PointLookupOptimizer.java 
> d83636d 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/pcr/PcrExprProcFactory.java 
> 825938a 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/OpProcFactory.java 
> 7262164 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java 6a31689 
>   ql/src/test/queries/clientpositive/pcs.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/pointlookup.q 1aef2ef 
>   ql/src/test/queries/clientpositive/pointlookup2.q 31bebbb 
>   ql/src/test/queries/clientpositive/pointlookup3.q 3daa94b 
>   ql/src/test/results/clientpositive/dynpart_sort_optimization_acid.q.out 
> eca29df 
>   ql/src/test/results/clientpositive/pcs.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/pointlookup.q.out 7e19be4 
>   ql/src/test/results/clientpositive/pointlookup2.q.out 55edd90 
>   ql/src/test/results/clientpositive/pointlookup3.q.out 4cfb97e 
> 
> Diff: https://reviews.apache.org/r/37778/diff/
> 
> 
> Testing
> ---
> 
> Local testing done. More unit tests coming in the next patch.
> 
> 
> Thanks,
> 
> Hari Sankar Sivarama Subramaniyan
> 
>

[jira] [Created] (HIVE-12061) add file type support to file metadata by expr call

2015-10-07 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-12061:
---

 Summary: add file type support to file metadata by expr call
 Key: HIVE-12061
 URL: https://issues.apache.org/jira/browse/HIVE-12061
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin


Expr filtering, automatic caching, etc. should be aware of file types for 
advanced features. For now only ORC is supported, but I want to add boundary 
for between ORC-specific and general metastore code, that could later be used 
for other formats if needed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-12063) Pad Decimal numbers with trailing zeros to the scale of the column

2015-10-07 Thread Xuefu Zhang (JIRA) 
Xuefu Zhang created HIVE-12063:
--

 Summary: Pad Decimal numbers with trailing zeros to the scale of 
the column
 Key: HIVE-12063
 URL: https://issues.apache.org/jira/browse/HIVE-12063
 Project: Hive
  Issue Type: Improvement
  Components: Types
Affects Versions: 1.1.0, 1.2.0, 1.0.0, 0.14.0, 0.13
Reporter: Xuefu Zhang
Assignee: Xuefu Zhang


HIVE-7373 was to address the problem of trimming tailing zeros by Hive, which 
caused many problems including treating 0.0, 0.00 and so on as 0, which has 
different precision/scale. Please refer to HIVE-7373 description. However, 
HIVE-7373 was reverted by HIVE-8745 while the underlying problems remained. 
HIVE-11835 was resolved recently to address one of the problems, where 0.0, 
0.00, and so cannot be read into decimal(1,1).

However, HIVE-11835 didn't address the problem of showing as 0 in query result 
for any decimal values such as 0.0, 0.00, etc. This causes confusion as 0 and 
0.0 have different precision/scale than 0.

The proposal here is to pad zeros for query result to the type's scale. This 
not only removes the confusion described above, but also aligns with many other 
DBs. Internal decimal number representation doesn't change, however.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-12057) ORC sarg is logged too much

2015-10-07 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-12057:
---

 Summary: ORC sarg is logged too much
 Key: HIVE-12057
 URL: https://issues.apache.org/jira/browse/HIVE-12057
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Priority: Minor
 Attachments: HIVE-12057.patch

SARG itself has too many newlines and it's logged for every splitgenerator in 
split generation



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-12062) enable HBase metastore file metadata cache for tez tests

2015-10-07 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-12062:
---

 Summary: enable HBase metastore file metadata cache for tez tests
 Key: HIVE-12062
 URL: https://issues.apache.org/jira/browse/HIVE-12062
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 37778: HIVE-11634

2015-10-07 Thread Hari Sankar Sivarama Subramaniyan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37778/
---

(Updated Oct. 7, 2015, 10:57 p.m.)


Review request for hive, Ashutosh Chauhan, Jesús Camacho Rodríguez, and John 
Pullokkaran.


Repository: hive-git


Description
---

Support partition pruning for IN(STRUCT(partcol, nonpartcol..)...)


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java bf48f69 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 439f616 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/PartitionColumnsSeparator.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/PointLookupOptimizer.java 
d83636d 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/pcr/PcrExprProcFactory.java 
825938a 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/OpProcFactory.java 
7262164 
  ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java 6a31689 
  ql/src/test/queries/clientpositive/pcs.q PRE-CREATION 
  ql/src/test/queries/clientpositive/pointlookup.q 1aef2ef 
  ql/src/test/queries/clientpositive/pointlookup2.q 31bebbb 
  ql/src/test/queries/clientpositive/pointlookup3.q 3daa94b 
  ql/src/test/results/clientpositive/dynpart_sort_optimization_acid.q.out 
eca29df 
  ql/src/test/results/clientpositive/pcs.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/pointlookup.q.out 7e19be4 
  ql/src/test/results/clientpositive/pointlookup2.q.out 55edd90 
  ql/src/test/results/clientpositive/pointlookup3.q.out 4cfb97e 

Diff: https://reviews.apache.org/r/37778/diff/


Testing
---

Local testing done. More unit tests coming in the next patch.


Thanks,

Hari Sankar Sivarama Subramaniyan

[jira] [Created] (HIVE-12060) LLAP: create separate variable for llap tests

2015-10-07 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-12060:
---

 Summary: LLAP: create separate variable for llap tests
 Key: HIVE-12060
 URL: https://issues.apache.org/jira/browse/HIVE-12060
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin


No real reason to just reuse tez one



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 37778: HIVE-11634

2015-10-07 Thread John Pullokkaran 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37778/#review101814
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/pcr/PcrExprProcFactory.java 
(line 385)


Add unit test cases for:
1. DNF(CNF)
expr1 or expr2 (where expr2 is conjuctine tree)
2. In clause involving part/non-part col that are combined by disjunction


- John Pullokkaran


On Oct. 7, 2015, 7:47 p.m., Hari Sankar Sivarama Subramaniyan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/37778/
> ---
> 
> (Updated Oct. 7, 2015, 7:47 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, Jesús Camacho Rodríguez, and John 
> Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Support partition pruning for IN(STRUCT(partcol, nonpartcol..)...)
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 54a529e 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 439f616 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/PartitionColumnsSeparator.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/PointLookupOptimizer.java 
> d83636d 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/pcr/PcrExprProcFactory.java 
> 825938a 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/OpProcFactory.java 
> 7262164 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java 6a31689 
>   ql/src/test/queries/clientpositive/pcs.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/pointlookup.q 1aef2ef 
>   ql/src/test/queries/clientpositive/pointlookup2.q 31bebbb 
>   ql/src/test/queries/clientpositive/pointlookup3.q 3daa94b 
>   ql/src/test/results/clientpositive/dynpart_sort_optimization_acid.q.out 
> eca29df 
>   ql/src/test/results/clientpositive/pcs.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/pointlookup.q.out 7e19be4 
>   ql/src/test/results/clientpositive/pointlookup2.q.out 55edd90 
>   ql/src/test/results/clientpositive/pointlookup3.q.out 4cfb97e 
> 
> Diff: https://reviews.apache.org/r/37778/diff/
> 
> 
> Testing
> ---
> 
> Local testing done. More unit tests coming in the next patch.
> 
> 
> Thanks,
> 
> Hari Sankar Sivarama Subramaniyan
> 
>

[jira] [Created] (HIVE-12065) FS stats collection may generate incorrect stats for multi-insert query

2015-10-07 Thread Ashutosh Chauhan (JIRA) 
Ashutosh Chauhan created HIVE-12065:
---

 Summary: FS stats collection may generate incorrect stats for 
multi-insert query
 Key: HIVE-12065
 URL: https://issues.apache.org/jira/browse/HIVE-12065
 Project: Hive
  Issue Type: Bug
  Components: Statistics
Affects Versions: 1.1.0, 1.2.0, 1.0.0, 0.14.0
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Schema evolution for parquet table

2015-10-07 Thread Mohammad Islam 


Hi Sergio,
Thanks for your reply.

I found one such effort : 
https://issues.apache.org/jira/browse/HIVE-6784

I consider to try it differently.
Similar stuffs in ORC:
https://issues.apache.org/jira/browse/HIVE-10591


Regards,
Mohammad


On Wednesday, October 7, 2015 8:58 AM, Sergio Pena  
wrote:
Hi Mohammad,

Currently, Hive + Parquet does not support auto casting for wider types.
That is be a very good idea to implement in Hive.
I'll investigate the hive + parquet code, and see if it is something we can
add in a future release.

- Sergio



On Tue, Oct 6, 2015 at 7:23 PM, Mohammad Islam 
wrote:

>
> Any hive+parquet user/dev to address this?
>
>
> Regards,
> Mohammad
>
> On Monday, October 5, 2015 3:41 PM, Mohammad Islam 
> wrote:
>
>
>
> Hi,
> Does the parquet table support auto casting to wider data types? For
> example, If I have a parquet table where some parquet data files which have
> "int"  as data type and other files have "long" data type for the same
> field.
>
> The table schema has type "bigint" for the same field.
> Does hive can read the file that was written with type "int"?
>
> I got this exception "Failed with exception
> java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException: org.apache.hadoop.io.IntWritable cannot be
> cast to org.apache.hadoop.io.LongWritable".
>
> Regards,
> Mohammad
>

[jira] [Created] (HIVE-12066) Add javadoc for methods added to public APIs

2015-10-07 Thread Owen O'Malley (JIRA) 
Owen O'Malley created HIVE-12066:


 Summary: Add javadoc for methods added to public APIs
 Key: HIVE-12066
 URL: https://issues.apache.org/jira/browse/HIVE-12066
 Project: Hive
  Issue Type: Sub-task
Reporter: Owen O'Malley
Assignee: Sergey Shelukhin


Looking through the changes for ORC, there are methods being added without 
documentation:

{code}
--- ql/src/java/org/apache/hadoop/hive/ql/io/orc/Reader.java
+++ ql/src/java/org/apache/hadoop/hive/ql/io/orc/Reader.java
@@ -360,8 +353,18 @@ RecordReader rows(long offset, long length,

   MetadataReader metadata() throws IOException;

+  List getVersionList();
+
+  int getMetadataSize();
+
+  List getOrcProtoStripeStatistics();
+
+  List getStripeStatistics();
+
+  List getOrcProtoFileStatistics();
+
+  DataReader createDefaultDataReader(boolean useZeroCopy);
+
{code}

You really need to look through all of the interfaces and fix them before 
merging into master.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 37778: HIVE-11634

2015-10-07 Thread John Pullokkaran 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37778/#review101840
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/pcr/PcrExprProcFactory.java 
(line 392)


This too should be unknown right?


- John Pullokkaran


On Oct. 7, 2015, 10:57 p.m., Hari Sankar Sivarama Subramaniyan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/37778/
> ---
> 
> (Updated Oct. 7, 2015, 10:57 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, Jesús Camacho Rodríguez, and John 
> Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Support partition pruning for IN(STRUCT(partcol, nonpartcol..)...)
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java bf48f69 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 439f616 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/PartitionColumnsSeparator.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/PointLookupOptimizer.java 
> d83636d 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/pcr/PcrExprProcFactory.java 
> 825938a 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/OpProcFactory.java 
> 7262164 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java 6a31689 
>   ql/src/test/queries/clientpositive/pcs.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/pointlookup.q 1aef2ef 
>   ql/src/test/queries/clientpositive/pointlookup2.q 31bebbb 
>   ql/src/test/queries/clientpositive/pointlookup3.q 3daa94b 
>   ql/src/test/results/clientpositive/dynpart_sort_optimization_acid.q.out 
> eca29df 
>   ql/src/test/results/clientpositive/pcs.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/pointlookup.q.out 7e19be4 
>   ql/src/test/results/clientpositive/pointlookup2.q.out 55edd90 
>   ql/src/test/results/clientpositive/pointlookup3.q.out 4cfb97e 
> 
> Diff: https://reviews.apache.org/r/37778/diff/
> 
> 
> Testing
> ---
> 
> Local testing done. More unit tests coming in the next patch.
> 
> 
> Thanks,
> 
> Hari Sankar Sivarama Subramaniyan
> 
>

Re: Review Request 38735: CBO: Calcite Operator To Hive Operator (Calcite Return Path): correct table column name in CTAS queries

2015-10-07 Thread pengcheng xiong 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38735/
---

(Updated Oct. 8, 2015, 12:11 a.m.)


Review request for hive and Ashutosh Chauhan.


Repository: hive-git


Description
---

To repro, run lineage2.q with return path turned on.


Diffs (updated)
-

  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcDriver2.java 
3aa6bce 
  
itests/hive-unit/src/test/java/org/apache/hive/jdbc/cbo_rp_TestJdbcDriver2.java 
PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/PlanModifierForASTConv.java
 16a375c 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 9c731b8 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 7a54aec 
  ql/src/test/queries/clientpositive/cbo_rp_lineage2.q PRE-CREATION 
  ql/src/test/results/clientpositive/cbo_rp_lineage2.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/38735/diff/


Testing
---


Thanks,

pengcheng xiong

[jira] [Created] (HIVE-12064) prevent transactional=false

2015-10-07 Thread Eugene Koifman (JIRA) 
Eugene Koifman created HIVE-12064:
-

 Summary: prevent transactional=false
 Key: HIVE-12064
 URL: https://issues.apache.org/jira/browse/HIVE-12064
 Project: Hive
  Issue Type: Bug
  Components: Transactions
Affects Versions: 1.0.0
Reporter: Eugene Koifman
Assignee: Eugene Koifman


currently a tblproperty transactional=true must be set to make a table behave 
in ACID compliant way.

This is misleading in that it seems like changing it to transactional=false 
makes the table non-acid but on disk layout of acid table is different than 
plain tables.  So changing this  property may cause wrong data to be returned.

Should prevent transactional=false.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 38211: HIVE-11684

2015-10-07 Thread John Pullokkaran 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38211/#review101851
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java
 (line 80)


"reduction" as it is set is not a %.
Shouldn't you multiply by ((100-reduction)/100)

guarding for reduction to be <= 100.


- John Pullokkaran


On Sept. 28, 2015, 11:40 a.m., Jesús Camacho Rodríguez wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38211/
> ---
> 
> (Updated Sept. 28, 2015, 11:40 a.m.)
> 
> 
> Review request for hive and John Pullokkaran.
> 
> 
> Bugs: HIVE-11684
> https://issues.apache.org/jira/browse/HIVE-11684
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Implement limit pushdown through outer join in CBO
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 
> dffdb5c14114d9c2d1e4c44a36f6b60966dcfc86 
>   itests/src/test/resources/testconfiguration.properties 
> 4f7b25f2563683933b9d677626f0447705927966 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortProjectTransposeRule.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 
> 9c731b8c5e58570570812b525b8bf0edb0f4c0a9 
>   ql/src/test/queries/clientpositive/limit_join_transpose.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q PRE-CREATION 
>   ql/src/test/results/clientpositive/limit_join_transpose.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_3.q.out 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/38211/diff/
> 
> 
> Testing
> ---
> 
> Two new tests added: limit_join_transpose.q and tez_dynpart_hashjoin_3.q 
> (only Tez)
> 
> 
> Thanks,
> 
> Jesús Camacho Rodríguez
> 
>

[jira] [Created] (HIVE-12055) Create row-by-row shims for the write path

2015-10-07 Thread Owen O'Malley (JIRA) 
Owen O'Malley created HIVE-12055:


 Summary: Create row-by-row shims for the write path 
 Key: HIVE-12055
 URL: https://issues.apache.org/jira/browse/HIVE-12055
 Project: Hive
  Issue Type: Sub-task
Reporter: Owen O'Malley
Assignee: Owen O'Malley


As part of removing the row-by-row writer, we'll need to shim out the higher 
level API (OrcSerde and OrcOutputFormat) so that we maintain backwards 
compatibility.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 37778: HIVE-11634

2015-10-07 Thread Hari Sankar Sivarama Subramaniyan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37778/
---

(Updated Oct. 7, 2015, 7:47 p.m.)


Review request for hive, Ashutosh Chauhan, Jesús Camacho Rodríguez, and John 
Pullokkaran.


Repository: hive-git


Description
---

Support partition pruning for IN(STRUCT(partcol, nonpartcol..)...)


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 54a529e 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 439f616 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/PartitionColumnsSeparator.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/PointLookupOptimizer.java 
d83636d 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/pcr/PcrExprProcFactory.java 
825938a 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/OpProcFactory.java 
7262164 
  ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java 6a31689 
  ql/src/test/queries/clientpositive/pcs.q PRE-CREATION 
  ql/src/test/queries/clientpositive/pointlookup.q 1aef2ef 
  ql/src/test/queries/clientpositive/pointlookup2.q 31bebbb 
  ql/src/test/queries/clientpositive/pointlookup3.q 3daa94b 
  ql/src/test/results/clientpositive/dynpart_sort_optimization_acid.q.out 
eca29df 
  ql/src/test/results/clientpositive/pcs.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/pointlookup.q.out 7e19be4 
  ql/src/test/results/clientpositive/pointlookup2.q.out 55edd90 
  ql/src/test/results/clientpositive/pointlookup3.q.out 4cfb97e 

Diff: https://reviews.apache.org/r/37778/diff/


Testing
---

Local testing done. More unit tests coming in the next patch.


Thanks,

Hari Sankar Sivarama Subramaniyan

Re: Review Request 37778: HIVE-11634

2015-10-07 Thread Hari Sankar Sivarama Subramaniyan 


> On Oct. 7, 2015, 6:21 p.m., John Pullokkaran wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/pcr/PcrExprProcFactory.java,
> >  line 377
> > 
> >
> > 1. We shouldn't remove non deterministic UDFs.
> > 2. Also make sure that partitionPruner isn't considering non 
> > deterministic UDFs
> > 3. Please add testcase that involves non deterministic udf
> > 4. Add a test case involving conditional expressions (case, If..)
> > Ex: select key from t1 where (case when key=1 then 10 else 20 end) in 
> > (10,11);

Thanks for reviewing the patch, John
1, 3, 4. Done in the new patch
2. Verified in the existing code base


- Hari Sankar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37778/#review101804
---


On Oct. 7, 2015, 7:47 p.m., Hari Sankar Sivarama Subramaniyan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/37778/
> ---
> 
> (Updated Oct. 7, 2015, 7:47 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, Jesús Camacho Rodríguez, and John 
> Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Support partition pruning for IN(STRUCT(partcol, nonpartcol..)...)
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 54a529e 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 439f616 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/PartitionColumnsSeparator.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/PointLookupOptimizer.java 
> d83636d 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/pcr/PcrExprProcFactory.java 
> 825938a 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/OpProcFactory.java 
> 7262164 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java 6a31689 
>   ql/src/test/queries/clientpositive/pcs.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/pointlookup.q 1aef2ef 
>   ql/src/test/queries/clientpositive/pointlookup2.q 31bebbb 
>   ql/src/test/queries/clientpositive/pointlookup3.q 3daa94b 
>   ql/src/test/results/clientpositive/dynpart_sort_optimization_acid.q.out 
> eca29df 
>   ql/src/test/results/clientpositive/pcs.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/pointlookup.q.out 7e19be4 
>   ql/src/test/results/clientpositive/pointlookup2.q.out 55edd90 
>   ql/src/test/results/clientpositive/pointlookup3.q.out 4cfb97e 
> 
> Diff: https://reviews.apache.org/r/37778/diff/
> 
> 
> Testing
> ---
> 
> Local testing done. More unit tests coming in the next patch.
> 
> 
> Thanks,
> 
> Hari Sankar Sivarama Subramaniyan
> 
>

[jira] [Created] (HIVE-12056) Branch 1.1.1: root pom and itest pom are not linked

2015-10-07 Thread Vaibhav Gumashta (JIRA) 
Vaibhav Gumashta created HIVE-12056:
---

 Summary: Branch 1.1.1: root pom and itest pom are not linked
 Key: HIVE-12056
 URL: https://issues.apache.org/jira/browse/HIVE-12056
 Project: Hive
  Issue Type: Bug
  Components: Testing Infrastructure
Affects Versions: 1.1.1
Reporter: Vaibhav Gumashta
Assignee: Vaibhav Gumashta






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: HIVE-TRUNK-JAVA8 #120

2015-10-07 Thread hiveqa 
See 


Changes:

[pxiong] HIVE-11699: Support special characters in quoted table names 
(Pengcheng Xiong, reviewed by John Pullokkaran)

[Xuefu Zhang] HIVE-11835: Type decimal(1,1) reads 0.0, 0.00, etc from text file 
as NULL (Reviewed by Szehon)

[Xuefu Zhang] HIVE-11835: Type decimal(1,1) reads 0.0, 0.00, etc from text file 
as NULL (Reviewed by Szehon)

[Sergio Pena] HIVE-12002: correct implementation typo (Alex Moundalexis, 
reviewed by Sergio Pena)

[Szehon Ho] HIVE-12007 : Hive LDAP Authenticator should allow just Domain 
without baseDN (for AD) (Naveen Gangam via Szehon)

[Szehon Ho] HIVE-11866 : Add framework to enable testing using LDAPServer using 
LDAP protocol (Naveen Gangam via Szehon)

[ekoifman] HIVE-11983 - Hive streaming API uses incorrect logic to assign 
buckets to incoming records (Roshan Naik via Eugene Koifman)

[vgumashta] HIVE-11720: Allow HiveServer2 to set custom http request/response 
header size (Vaibhav Gumashta reviewed by Thejas Nair)

[sershe] HIVE-12027 : simplify file metadata cache ppd api (Sergey Shelukhin, 
reviewed by Alan Gates)

[ctang] HIVE-11786: Deprecate the use of redundant column in colunm stats 
related tables (Chaoyu Tang, reviewed by Szehon Ho, Sergey Shelukhin)

[pxiong] HIVE-12011: unable to create temporary table using CTAS if regular 
table with that name already exists (Pengcheng Xiong, reviewed by Laljo John 
Pullokkaran)

--
[...truncated 324 lines...]
Aptest2/src/main/java/org/apache/hive/ptest/execution/JIRAService.java
Aptest2/src/main/java/org/apache/hive/ptest/execution/Drone.java
Aptest2/src/main/java/org/apache/hive/ptest/execution/Constants.java
Aptest2/src/main/java/org/apache/hive/ptest/execution/ssh
A
ptest2/src/main/java/org/apache/hive/ptest/execution/ssh/RemoteCommandResult.java
A
ptest2/src/main/java/org/apache/hive/ptest/execution/ssh/SSHCommandExecutor.java
A
ptest2/src/main/java/org/apache/hive/ptest/execution/ssh/RSyncCommandExecutor.java
Aptest2/src/main/java/org/apache/hive/ptest/execution/ssh/SSHCommand.java
Aptest2/src/main/java/org/apache/hive/ptest/execution/ssh/SSHResult.java
Aptest2/src/main/java/org/apache/hive/ptest/execution/ssh/RSyncCommand.java
Aptest2/src/main/java/org/apache/hive/ptest/execution/ssh/RSyncResult.java
A
ptest2/src/main/java/org/apache/hive/ptest/execution/ssh/NonZeroExitCodeException.java
A
ptest2/src/main/java/org/apache/hive/ptest/execution/ssh/AbstractSSHCommand.java
A
ptest2/src/main/java/org/apache/hive/ptest/execution/ssh/AbstractSSHResult.java
A
ptest2/src/main/java/org/apache/hive/ptest/execution/ssh/SSHExecutionException.java
A
ptest2/src/main/java/org/apache/hive/ptest/execution/AbortDroneException.java
A
ptest2/src/main/java/org/apache/hive/ptest/execution/LocalCommandFactory.java
Aptest2/src/main/java/org/apache/hive/ptest/api
Aptest2/src/main/java/org/apache/hive/ptest/api/Status.java
Aptest2/src/main/java/org/apache/hive/ptest/api/server
Aptest2/src/main/java/org/apache/hive/ptest/api/server/TestExecutor.java
Aptest2/src/main/java/org/apache/hive/ptest/api/server/Test.java
Aptest2/src/main/java/org/apache/hive/ptest/api/server/TestLogger.java
A
ptest2/src/main/java/org/apache/hive/ptest/api/server/ExecutionController.java
Aptest2/src/main/java/org/apache/hive/ptest/api/request
Aptest2/src/main/java/org/apache/hive/ptest/api/request/TestLogRequest.java
A
ptest2/src/main/java/org/apache/hive/ptest/api/request/TestStatusRequest.java
Aptest2/src/main/java/org/apache/hive/ptest/api/request/TestStopRequest.java
Aptest2/src/main/java/org/apache/hive/ptest/api/request/TestListRequest.java
A
ptest2/src/main/java/org/apache/hive/ptest/api/request/TestStartRequest.java
Aptest2/src/main/java/org/apache/hive/ptest/api/client
Aptest2/src/main/java/org/apache/hive/ptest/api/client/PTestClient.java
Aptest2/src/main/java/org/apache/hive/ptest/api/response
A
ptest2/src/main/java/org/apache/hive/ptest/api/response/TestLogResponse.java
A
ptest2/src/main/java/org/apache/hive/ptest/api/response/TestStatusResponse.java
Aptest2/src/main/java/org/apache/hive/ptest/api/response/TestStatus.java
A
ptest2/src/main/java/org/apache/hive/ptest/api/response/TestStopResponse.java
A
ptest2/src/main/java/org/apache/hive/ptest/api/response/TestListResponse.java
A
ptest2/src/main/java/org/apache/hive/ptest/api/response/GenericResponse.java
A
ptest2/src/main/java/org/apache/hive/ptest/api/response/TestStartResponse.java
Aptest2/pom.xml
Aptest2/README.md
Checked out revision 1707219.
+ cd ptest2
+ mvn clean package -DskipTests -Drat.numUnapprovedLicenses=1000 
-Dmaven.repo.local=
[INFO] Scanning for projects...
[INFO]

[jira] [Created] (HIVE-12053) Stats performance regression caused by HIVE-11786

2015-10-07 Thread Chaoyu Tang (JIRA) 
Chaoyu Tang created HIVE-12053:
--

 Summary: Stats performance regression caused by HIVE-11786
 Key: HIVE-12053
 URL: https://issues.apache.org/jira/browse/HIVE-12053
 Project: Hive
  Issue Type: Bug
  Components: Metastore
Reporter: Chaoyu Tang


HIVE-11786 tried to normalize table TAB_COL_STATS/PART_COL_STATS but caused 
performance regression.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Schema evolution for parquet table

2015-10-07 Thread Sergio Pena 
Hi Mohammad,

Currently, Hive + Parquet does not support auto casting for wider types.
That is be a very good idea to implement in Hive.
I'll investigate the hive + parquet code, and see if it is something we can
add in a future release.

- Sergio


On Tue, Oct 6, 2015 at 7:23 PM, Mohammad Islam 
wrote:

>
> Any hive+parquet user/dev to address this?
>
>
> Regards,
> Mohammad
>
> On Monday, October 5, 2015 3:41 PM, Mohammad Islam 
> wrote:
>
>
>
> Hi,
> Does the parquet table support auto casting to wider data types? For
> example, If I have a parquet table where some parquet data files which have
> "int"  as data type and other files have "long" data type for the same
> field.
>
> The table schema has type "bigint" for the same field.
> Does hive can read the file that was written with type "int"?
>
> I got this exception "Failed with exception
> java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException: org.apache.hadoop.io.IntWritable cannot be
> cast to org.apache.hadoop.io.LongWritable".
>
> Regards,
> Mohammad
>

Re: Review Request 37778: HIVE-11634

2015-10-07 Thread John Pullokkaran 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37778/#review101804
---



ql/src/java/org/apache/hadoop/hive/ql/optimizer/pcr/PcrExprProcFactory.java 
(line 377)


1. We shouldn't remove non deterministic UDFs.
2. Also make sure that partitionPruner isn't considering non deterministic 
UDFs
3. Please add testcase that involves non deterministic udf
4. Add a test case involving conditional expressions (case, If..)
Ex: select key from t1 where (case when key=1 then 10 else 20 end) in 
(10,11);


- John Pullokkaran


On Oct. 1, 2015, 11:42 p.m., Hari Sankar Sivarama Subramaniyan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/37778/
> ---
> 
> (Updated Oct. 1, 2015, 11:42 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, Jesús Camacho Rodríguez, and John 
> Pullokkaran.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Support partition pruning for IN(STRUCT(partcol, nonpartcol..)...)
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java dffdb5c 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 439f616 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/PartitionColumnsSeparator.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/PointLookupOptimizer.java 
> d83636d 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/pcr/PcrExprProcFactory.java 
> 825938a 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/OpProcFactory.java 
> 7262164 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java 6a31689 
>   ql/src/test/queries/clientpositive/pcs.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/pointlookup.q 1aef2ef 
>   ql/src/test/queries/clientpositive/pointlookup2.q 31bebbb 
>   ql/src/test/queries/clientpositive/pointlookup3.q 3daa94b 
>   ql/src/test/results/clientpositive/dynpart_sort_optimization_acid.q.out 
> eca29df 
>   ql/src/test/results/clientpositive/pcs.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/pointlookup.q.out 7e19be4 
>   ql/src/test/results/clientpositive/pointlookup2.q.out 55edd90 
>   ql/src/test/results/clientpositive/pointlookup3.q.out 4cfb97e 
> 
> Diff: https://reviews.apache.org/r/37778/diff/
> 
> 
> Testing
> ---
> 
> Local testing done. More unit tests coming in the next patch.
> 
> 
> Thanks,
> 
> Hari Sankar Sivarama Subramaniyan
> 
>

[jira] [Created] (HIVE-12054) Create vectorized write method

2015-10-07 Thread Owen O'Malley (JIRA) 
Owen O'Malley created HIVE-12054:


 Summary: Create vectorized write method
 Key: HIVE-12054
 URL: https://issues.apache.org/jira/browse/HIVE-12054
 Project: Hive
  Issue Type: Sub-task
  Components: File Formats
Reporter: Owen O'Malley
Assignee: Owen O'Malley


We need to add writer methods that can write VectorizedRowBatch to an ORC file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)