Re: Review Request 46174: HIVE-13509: HCatalog getSplits should ignore the partition with invalid path

2016-04-14 Thread Chaoyu Tang 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/46174/
---

(Updated April 15, 2016, 3:05 a.m.)


Review request for hive.


Changes
---

add a HCatlog property in order to keep the backwards compatibility


Bugs: HIVE-13509
https://issues.apache.org/jira/browse/HIVE-13509


Repository: hive-git


Description
---

Remove invalid path in HCatBaseInputFormat.getSplits.


Diffs (updated)
-

  
hcatalog/core/src/main/java/org/apache/hive/hcatalog/common/HCatConstants.java 
6b03fcb 
  
hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/HCatBaseInputFormat.java
 adfaf4e 
  
hcatalog/hcatalog-pig-adapter/src/test/java/org/apache/hive/hcatalog/pig/TestHCatLoader.java
 2440cb5 
  
hcatalog/hcatalog-pig-adapter/src/test/java/org/apache/hive/hcatalog/pig/TestHCatLoaderWithProps.java
 PRE-CREATION 

Diff: https://reviews.apache.org/r/46174/diff/


Testing
---

New unit test passed
Precommit tests


Thanks,

Chaoyu Tang

Re: Review Request 45911: HIVE-13341: Stats state is not captured correctly: differentiate load table and create table

2016-04-14 Thread pengcheng xiong 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45911/
---

(Updated April 14, 2016, 11:52 p.m.)


Review request for hive and Ashutosh Chauhan.


Repository: hive-git


Description
---

HIVE-13341: Stats state is not captured correctly: differentiate load table and 
create table


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java 41d150c 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/history/TestHiveHistory.java
 c046708 
  metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
76220f4 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 54592cc 
  ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java 87a7667 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 4c9acce 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java 
549d24f 
  ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java 03b4d8b 
  ql/src/test/org/apache/hadoop/hive/ql/exec/TestExecDriver.java eaeb66b 
  ql/src/test/queries/clientpositive/insert_values_orig_table_use_metadata.q 
PRE-CREATION 
  
ql/src/test/results/clientpositive/insert_values_orig_table_use_metadata.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/join32.q.out 5cb124b 
  ql/src/test/results/clientpositive/stats0.q.out 99326fc 
  ql/src/test/results/clientpositive/stats1.q.out 72c53e3 
  ql/src/test/results/clientpositive/stats10.q.out d7519af 
  ql/src/test/results/clientpositive/stats11.q.out 7598dc0 
  ql/src/test/results/clientpositive/stats12.q.out 6a44d15 
  ql/src/test/results/clientpositive/stats13.q.out f98753e 
  ql/src/test/results/clientpositive/stats16.q.out 3b371a8 
  ql/src/test/results/clientpositive/stats18.q.out 6971e44 
  ql/src/test/results/clientpositive/stats20.q.out d7e52b4 
  ql/src/test/results/clientpositive/stats3.q.out 0d8cbbd 
  ql/src/test/results/clientpositive/stats5.q.out 74ddadb 
  ql/src/test/results/clientpositive/stats6.q.out f2a756f 
  ql/src/test/results/clientpositive/stats_noscan_1.q.out 7382b31 
  ql/src/test/results/clientpositive/stats_partscan_1_23.q.out ab2baaa 

Diff: https://reviews.apache.org/r/45911/diff/


Testing
---


Thanks,

pengcheng xiong

Re: Review Request 45406: HIVE-13349: Metastore Changes : API calls for retrieving primary keys and foreign keys information

2016-04-14 Thread Ashutosh Chauhan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45406/#review129030
---




metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java (line 3267)


Instead of making this check, we should just try to persist the row in 
constraint table, it will fail in case there is a duplicate name, since its a 
part of primary key, in which case we pick another name.



metastore/src/model/org/apache/hadoop/hive/metastore/model/MConstraintPK.java 
(line 7)


I don't see this class getting used anywhere. In jdo mapping its seems you 
are using inner class PK which is defined in MConstraint. Do we need both?


- Ashutosh Chauhan


On April 14, 2016, 10:44 p.m., Hari Sankar Sivarama Subramaniyan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45406/
> ---
> 
> (Updated April 14, 2016, 10:44 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Metastore Changes : API calls for retrieving primary keys and foreign keys 
> information
> 
> 
> Diffs
> -
> 
>   metastore/if/hive_metastore.thrift 6a55962 
>   metastore/scripts/upgrade/derby/034-HIVE-13076.derby.sql PRE-CREATION 
>   metastore/scripts/upgrade/derby/hive-schema-2.1.0.derby.sql 42f4eb6 
>   metastore/scripts/upgrade/derby/upgrade-2.0.0-to-2.1.0.derby.sql a0bac3c 
>   metastore/scripts/upgrade/mssql/019-HIVE-13076.mssql.sql PRE-CREATION 
>   metastore/scripts/upgrade/mssql/hive-schema-2.1.0.mssql.sql cf5a662 
>   metastore/scripts/upgrade/mssql/upgrade-2.0.0-to-2.1.0.mssql.sql f25daf2 
>   metastore/scripts/upgrade/mysql/034-HIVE-13076.mysql.sql PRE-CREATION 
>   metastore/scripts/upgrade/mysql/hive-schema-2.1.0.mysql.sql 6fd3209 
>   metastore/scripts/upgrade/mysql/upgrade-2.0.0-to-2.1.0.mysql.sql e790636 
>   metastore/scripts/upgrade/oracle/034-HIVE-13076.oracle.sql PRE-CREATION 
>   metastore/scripts/upgrade/oracle/hive-schema-2.1.0.oracle.sql 774f6be 
>   metastore/scripts/upgrade/oracle/upgrade-2.0.0-to-2.1.0.oracle.sql 8368d08 
>   metastore/scripts/upgrade/postgres/033-HIVE-13076.postgres.sql PRE-CREATION 
>   metastore/scripts/upgrade/postgres/hive-schema-2.1.0.postgres.sql 7463a37 
>   metastore/scripts/upgrade/postgres/upgrade-2.0.0-to-2.1.0.postgres.sql 
> 6172407 
>   
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ForeignKeysRequest.java
>  PRE-CREATION 
>   
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ForeignKeysResponse.java
>  PRE-CREATION 
>   
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PrimaryKeysRequest.java
>  PRE-CREATION 
>   
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PrimaryKeysResponse.java
>  PRE-CREATION 
>   
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SQLForeignKey.java
>  PRE-CREATION 
>   
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SQLPrimaryKey.java
>  PRE-CREATION 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
> c9fadad 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 
> 64a26ac 
>   metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 
> 39cf927 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 
> 06e9f78 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
> ac293b9 
>   metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java e49f757 
>   metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 
> a73dbeb 
>   metastore/src/model/org/apache/hadoop/hive/metastore/model/MConstraint.java 
> PRE-CREATION 
>   
> metastore/src/model/org/apache/hadoop/hive/metastore/model/MConstraintPK.java 
> PRE-CREATION 
>   metastore/src/model/package.jdo 7385a13 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
>  94ca835 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
>  b108f95 
> 
> Diff: https://reviews.apache.org/r/45406/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Hari Sankar Sivarama Subramaniyan
> 
>

[jira] [Created] (HIVE-13523) Fix connection leak in ORC RecordReader and refactor for unit testing

2016-04-14 Thread Thomas Poepping (JIRA) 
Thomas Poepping created HIVE-13523:
--

 Summary: Fix connection leak in ORC RecordReader and refactor for 
unit testing
 Key: HIVE-13523
 URL: https://issues.apache.org/jira/browse/HIVE-13523
 Project: Hive
  Issue Type: Bug
  Components: ORC
Affects Versions: 2.0.0
Reporter: Thomas Poepping


In RecordReaderImpl, a MetadataReaderImpl object was being created (opening a 
file), but never closed, causing a leak. This change closes the Metadata object 
in RecordReaderImpl, and does substantial refactoring to make RecordReaderImpl 
testable:
 * Created DataReaderFactory and MetadataReaderFactory (plus default 
implementations) so that the create() methods can be mocked to verify that the 
objects are actually closed in RecordReaderImpl.close()
 * Created MetadataReaderProperties and DataReaderProperties to clean up 
argument lists, making code more readable
 * Created a builder() for RecordReaderImpl to make the code more readable
 * DataReader and MetadataReader now extend closeable (there was no reason for 
them not to in the first place) so I can use the guava Closer interface: 
http://docs.guava-libraries.googlecode.com/git/javadoc/com/google/common/io/Closer.html
 * Use the Closer interface to guarantee that regardless of if either close() 
call fails, both will be attempted (preventing further potential leaks)
 * Create builders for MetadataReaderProperties, DataReaderProperties, and 
RecordReaderImpl to help with code readability



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 45406: HIVE-13349: Metastore Changes : API calls for retrieving primary keys and foreign keys information

2016-04-14 Thread Hari Sankar Sivarama Subramaniyan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45406/
---

(Updated April 14, 2016, 10:44 p.m.)


Review request for hive and Ashutosh Chauhan.


Changes
---

1. Added enable/validate/rely bits to the metastore.
2. Validate fk name, pk names check for duplicates.
3. Generate constraint names if the user does not provide one.


Repository: hive-git


Description
---

Metastore Changes : API calls for retrieving primary keys and foreign keys 
information


Diffs (updated)
-

  metastore/if/hive_metastore.thrift 6a55962 
  metastore/scripts/upgrade/derby/034-HIVE-13076.derby.sql PRE-CREATION 
  metastore/scripts/upgrade/derby/hive-schema-2.1.0.derby.sql 42f4eb6 
  metastore/scripts/upgrade/derby/upgrade-2.0.0-to-2.1.0.derby.sql a0bac3c 
  metastore/scripts/upgrade/mssql/019-HIVE-13076.mssql.sql PRE-CREATION 
  metastore/scripts/upgrade/mssql/hive-schema-2.1.0.mssql.sql cf5a662 
  metastore/scripts/upgrade/mssql/upgrade-2.0.0-to-2.1.0.mssql.sql f25daf2 
  metastore/scripts/upgrade/mysql/034-HIVE-13076.mysql.sql PRE-CREATION 
  metastore/scripts/upgrade/mysql/hive-schema-2.1.0.mysql.sql 6fd3209 
  metastore/scripts/upgrade/mysql/upgrade-2.0.0-to-2.1.0.mysql.sql e790636 
  metastore/scripts/upgrade/oracle/034-HIVE-13076.oracle.sql PRE-CREATION 
  metastore/scripts/upgrade/oracle/hive-schema-2.1.0.oracle.sql 774f6be 
  metastore/scripts/upgrade/oracle/upgrade-2.0.0-to-2.1.0.oracle.sql 8368d08 
  metastore/scripts/upgrade/postgres/033-HIVE-13076.postgres.sql PRE-CREATION 
  metastore/scripts/upgrade/postgres/hive-schema-2.1.0.postgres.sql 7463a37 
  metastore/scripts/upgrade/postgres/upgrade-2.0.0-to-2.1.0.postgres.sql 
6172407 
  
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ForeignKeysRequest.java
 PRE-CREATION 
  
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ForeignKeysResponse.java
 PRE-CREATION 
  
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PrimaryKeysRequest.java
 PRE-CREATION 
  
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PrimaryKeysResponse.java
 PRE-CREATION 
  
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SQLForeignKey.java
 PRE-CREATION 
  
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SQLPrimaryKey.java
 PRE-CREATION 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
c9fadad 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 
64a26ac 
  metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 
39cf927 
  metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 
06e9f78 
  metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java ac293b9 
  metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java e49f757 
  metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java 
a73dbeb 
  metastore/src/model/org/apache/hadoop/hive/metastore/model/MConstraint.java 
PRE-CREATION 
  metastore/src/model/org/apache/hadoop/hive/metastore/model/MConstraintPK.java 
PRE-CREATION 
  metastore/src/model/package.jdo 7385a13 
  
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
 94ca835 
  
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
 b108f95 

Diff: https://reviews.apache.org/r/45406/diff/


Testing
---


Thanks,

Hari Sankar Sivarama Subramaniyan

[jira] [Created] (HIVE-13522) regexp_extract.q hangs on master

2016-04-14 Thread Ashutosh Chauhan (JIRA) 
Ashutosh Chauhan created HIVE-13522:
---

 Summary: regexp_extract.q hangs on master
 Key: HIVE-13522
 URL: https://issues.apache.org/jira/browse/HIVE-13522
 Project: Hive
  Issue Type: Bug
  Components: Tests
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan


Disable to unblock Hive QA runs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13521) Add a generic timeout to non CliDriver tests

2016-04-14 Thread Siddharth Seth (JIRA) 
Siddharth Seth created HIVE-13521:
-

 Summary: Add a generic timeout to non CliDriver tests
 Key: HIVE-13521
 URL: https://issues.apache.org/jira/browse/HIVE-13521
 Project: Hive
  Issue Type: Sub-task
Reporter: Siddharth Seth
Assignee: Siddharth Seth


HIVE-13495 is adding a timeout to clidriver tests.

Add a timeout to all other tests as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13520) Don't allow any test to run for longer than 1 hour in the ptest setup

2016-04-14 Thread Siddharth Seth (JIRA) 
Siddharth Seth created HIVE-13520:
-

 Summary: Don't allow any test to run for longer than 1 hour in the 
ptest setup
 Key: HIVE-13520
 URL: https://issues.apache.org/jira/browse/HIVE-13520
 Project: Hive
  Issue Type: Improvement
Reporter: Siddharth Seth
Assignee: Siddharth Seth


Current timeout for batches is 2hours. This needs to be lowered. 1hour may be 
too much as well. We can start with this, and reduce timeouts further.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13519) Allow LlapRecordReader to parse/output rows

2016-04-14 Thread Jason Dere (JIRA) 
Jason Dere created HIVE-13519:
-

 Summary: Allow LlapRecordReader to parse/output rows
 Key: HIVE-13519
 URL: https://issues.apache.org/jira/browse/HIVE-13519
 Project: Hive
  Issue Type: Sub-task
  Components: llap
Reporter: Jason Dere
Assignee: Jason Dere
 Fix For: llap


Add a row interface for the LlapRecordReader, so the client doesn't have to 
parse the data it's getting from LLAP.
Also move away from using metastore Schema class, to avoid dependency on 
metastore.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13518) Hive on Tez: Shuffle joins do not choose the right 'big' table.

2016-04-14 Thread Vikram Dixit K (JIRA) 
Vikram Dixit K created HIVE-13518:
-

 Summary: Hive on Tez: Shuffle joins do not choose the right 'big' 
table.
 Key: HIVE-13518
 URL: https://issues.apache.org/jira/browse/HIVE-13518
 Project: Hive
  Issue Type: Bug
Reporter: Vikram Dixit K
Assignee: Vikram Dixit K


Currently the big table is always assumed to be at position 0 but this isn't 
efficient for some queries as the big table at position 1 could have a lot more 
keys/skew. We already have a mechanism of choosing the big table that can be 
leveraged to make the right choice.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13517) Hive logs in Spark Executor and Driver should show thread-id.

2016-04-14 Thread Szehon Ho (JIRA) 
Szehon Ho created HIVE-13517:


 Summary: Hive logs in Spark Executor and Driver should show 
thread-id.
 Key: HIVE-13517
 URL: https://issues.apache.org/jira/browse/HIVE-13517
 Project: Hive
  Issue Type: Bug
  Components: Spark
Affects Versions: 2.0.0, 1.2.1
Reporter: Szehon Ho
Assignee: Xuefu Zhang


In Spark, there might be more than one task running in one executor. Similarly, 
there may be more than one thread running in Driver.

This makes debugging through the logs a nightmare. It would be great if there 
could be thread-ids in the logs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13516) Adding BTEQ .IF, .QUIT, ERRORCODE to HPL/SQL

2016-04-14 Thread Dmitry Tolpeko (JIRA) 
Dmitry Tolpeko created HIVE-13516:
-

 Summary: Adding BTEQ .IF, .QUIT, ERRORCODE to HPL/SQL
 Key: HIVE-13516
 URL: https://issues.apache.org/jira/browse/HIVE-13516
 Project: Hive
  Issue Type: Improvement
Reporter: Dmitry Tolpeko
Assignee: Dmitry Tolpeko


Adding Teradata BTEQ features to HPL/SQL such as .IF, .QUIT, ERRORCODE.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Reviews & commits (RTC/CTR), contributions, bylaws

2016-04-14 Thread Lars Francke 
Okay I have started a VOTE thread on the user@ mailing list (as per the
bylaws). I would appreciate it if you could head over there and vote :)

Thank you!

On Thu, Apr 14, 2016 at 12:41 AM, Lars Francke 
wrote:

> Thanks for the +1 Alan.
>
> I agree that we're leaving potential contributions on the floor. Doing
> more reviews is definitely a very good step in the right direction. Thank
> you! I see this Bylaws change as another (small) step in the right
> direction. I'm sure we can come up with more ideas.
>
> I'll start a VOTE thread on the user@ mailing list.
>
> On Tue, Apr 12, 2016 at 5:32 PM, Alan Gates  wrote:
>
>> I’m +1 on this change of allowing simple cleanup changes without
>> requiring a full review.
>>
>> But jumping to this fix obscures a bigger problem we have as a
>> community.  This fix only works for committers, not for non-committers who
>> may also contribute such patches.  And it doesn’t solve the situation for
>> non-trivial patches.  We’re leaving potential contributions on the floor
>> and keeping people out of our community.  We need to solve this.
>>
>> One thing I’ve been doing over the last few months is set up a filter in
>> JIRA for components that I know well (metastore, acid, etc.) and then put a
>> recurring task in my task tracker app to review a patch every day.
>> Realistically I manage 2-3 reviews a week, but that’s 1-2 more than I was
>> doing before.  I encourage my fellow committers to find something that
>> works for them.  We need to improve the health of our community.
>>
>> Alan.
>>
>> > On Apr 12, 2016, at 07:56, Lars Francke  wrote:
>> >
>> > Thanks Thejas for the suggestion & others for jumping in. That seems
>> fine
>> > for me. 2 days also seems good. Holidays are different in almost every
>> > country so I wouldn't exclude those.
>> >
>> > I have followed the procedure used for the last Bylaws change and
>> created a
>> > new Wiki page here: <
>> >
>> https://cwiki.apache.org/confluence/display/Hive/Proposed+Changes+to+Hive+Project+Bylaws+-+April+2016
>> >> .
>> >
>> > It includes this paragraph: "Minor issues (e.g. typos, code style
>> issues,
>> > JavaDoc changes. At committer's discretion) can be committed after
>> > soliciting feedback/review on the mailing list and not receiving
>> feedback
>> > within 2 days."
>> > I'm not a native speaker so feedback is welcome.
>> >
>> > I also fixed three typos in the Bylaws (and marked them as changed): <
>> >
>> https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=62691925&selectedPageVersions=3&selectedPageVersions=2
>> >>
>> >
>> > Once the discussion settles down I'll open a vote thread on the user@
>> > mailing list which requires a 2/3 majority of all active PMC members. I
>> > couldn't find a definition of "active" though.
>> >
>> > On Mon, Apr 11, 2016 at 10:26 PM, Thejas Nair 
>> wrote:
>> >
>> >> I agree we have a problem here. At least patches as small as this
>> >> shouldn't take too long to get reviewed.
>> >>
>> >> Knox seems to consider a very large set of patches as being under CTR
>> >> process.
>> >> I think hive is very large and mature project that I would lean
>> >> towards RTC process for most issues. I think we can make an exception
>> >> for very minor patches such as fixing typos and and checkstyle issues.
>> >> Maybe the process can be to solicit reviews for such minor patches by
>> >> sending an email to dev@ list and if no response is seen in 2 days, go
>> >> ahead and commit it ?
>> >>
>> >>
>> >>
>> >>
>> >> On Mon, Apr 11, 2016 at 6:38 AM, Lars Francke 
>> >> wrote:
>> >>> Hi,
>> >>>
>> >>> I've been a long-time contributor to Hive (5 or so years) and have
>> been
>> >>> voted in as a committer and I'm very grateful for that. I also
>> understand
>> >>> that my situation is different than most or lots of committers as I'm
>> not
>> >>> working for one of the big companies (Facebook, Cloudera, Hortonworks
>> >> etc.)
>> >>> where you can just ask someone sitting next to you to do a review.
>> >>>
>> >>> I'd really like to contribute more than I do currently but the
>> process of
>> >>> getting patches in is painful for me (and other 'outside'
>> contributors)
>> >> as
>> >>> it is hard to get reviews & things committed. The nature of most of my
>> >>> patches is very minor[1] (fixing typos, checkstyle issues etc.) and I
>> >>> understand that these are not the most interesting patches to review
>> and
>> >>> are easy to miss. I don't blame anyone for this situation as I totally
>> >>> understand it and have been on the other side of this for other
>> projects.
>> >>>
>> >>> Is there anything we can do to make it easier for me and others like
>> me
>> >> to
>> >>> contribute here? I absolutely see the value in having "cleaner" code
>> and
>> >>> when done in small batches it's usually not very disruptive either.
>> >>>
>> >>> The bylaws currently require a +1 from a committer who has not
>> authored
>> >> the
>> >>> patch. Knox for example has a different policy

[jira] [Created] (HIVE-13515) Semantics of TypeInfoUtils.areColTypesCompatible changed which also changed the semantics of hive.metastore.disallow.incompatible.col.type.changes of

2016-04-14 Thread Matt McCline (JIRA) 
Matt McCline created HIVE-13515:
---

 Summary: Semantics of TypeInfoUtils.areColTypesCompatible changed 
which also changed the semantics of 
hive.metastore.disallow.incompatible.col.type.changes of 
 Key: HIVE-13515
 URL: https://issues.apache.org/jira/browse/HIVE-13515
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 2.1.0
Reporter: Matt McCline
Assignee: Ashutosh Chauhan
Priority: Critical



I had this test case in my new Schema Evolution patch (not checked in yet) that 
verifies STRING_GROUP to DECIMAL conversion:

{code}
-- SUBSECTION: ALTER TABLE CHANGE COLUMNS for STRING_GROUP -> DECIMAL: (STRING, 
CHAR, VARCHAR)
--
CREATE TABLE table_change_string_group_decimal(insert_num int, c1 STRING, c2 
CHAR(50), c3 VARCHAR(50), b STRING);

insert into table table_change_string_group_decimal
values (1, '753.7028', '753.7028', '753.7028', 'original'),
   (2, '-3651.672121', '-3651.672121', '-3651.672121', 'original'),
   (3, '-29.0764', '-29.0764', '-29.0764', 'original'),
   (4, '-10.3', '-10.3', '-10.3', 'original');

-- Table-Non-Cascade CHANGE COLUMNS ...
alter table table_change_string_group_decimal replace columns (insert_num int, 
c1 decimal(38,18), c2 decimal(38,18), c3 decimal(38,18), b STRING);

insert into table table_change_string_group_decimal
 values (5, 30.774, 30.774, 30.774, 'new'),
(6, 20.31, 20.31, 20.31, 'new'),
(7, 46114.284799488, 46114.284799488, 46114.284799488, 'new'),
(8, -66475.561431, -66475.561431, -66475.561431, 'new');

insert into table table_change_string_group_decimal
 values (9, 17808.963785, 17808.963785, 17808.963785, 'new'),
(10, 9250340.75 , 9250340.75 , 9250340.75 , 'new');

explain
select insert_num,c1,c2,c3,b from table_change_string_group_decimal order by 
insert_num;

select insert_num,c1,c2,c3,b from table_change_string_group_decimal order by 
insert_num;

drop table table_change_string_group_decimal;
{code}

that now fails on master (I suspect because of HIVE-13380 Decimal should have 
lower precedence than double in type hierachy) when:

{code}
set hive.metastore.disallow.incompatible.col.type.changes=true;
{code}

with this call stack:
{code}
Caused by: InvalidOperationException(message:The following columns have types 
incompatible with the existing columns in their respective positions :
c1,c2,c3)
at 
org.apache.hadoop.hive.metastore.MetaStoreUtils.throwExceptionIfIncompatibleColTypeChange(MetaStoreUtils.java:599)
at 
org.apache.hadoop.hive.metastore.HiveAlterHandler.alterTable(HiveAlterHandler.java:132)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_core(HiveMetaStore.java:3502)
at 
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_table_with_environment_context(HiveMetaStore.java:3473)
{code}

Key point is MetaStoreUtils.throwExceptionIfIncompatibleColTypeChange is 
calling TypeInfoUtils.areColTypesCompatible which changed

-// Allow implicit String to Decimal conversion
-if (fromPg == PrimitiveGrouping.STRING_GROUP && to == 
PrimitiveCategory.DECIMAL) {
-  return true;
-}

Will not current customers that have scripts with 
hive.metastore.disallow.incompatible.col.type.changes=true that do Schema 
Evolution get bitten?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)