Review Request 62321: HIVE-17530: ClassCastException when converting uniontype

[Anthony Hsu via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62321/
---

Review request for hive, Carl Steinbach and Ratandeep Ratti.


Bugs: HIVE-17530
https://issues.apache.org/jira/browse/HIVE-17530


Repository: hive-git


Description
---

Previously, StandardUnionObjectInspector was creating an ArrayList instead of a 
StandardUnion, causing the exception

```
java.lang.ClassCastException: java.util.ArrayList cannot be cast to 
org.apache.hadoop.hive.serde2.objectinspector.UnionObject
```

This patch fixes this.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorDeserializeRow.java 
2ad06fc12869e74e14aae7b7a36685482c4a1ade 
  ql/src/test/queries/clientpositive/orc_avro_partition_uniontype.q 
PRE-CREATION 
  ql/src/test/results/clientpositive/orc_avro_partition_uniontype.q.out 
PRE-CREATION 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorConverters.java
 7921de8d9c4a56af715de5498954794aaba32fff 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/SettableUnionObjectInspector.java
 564d8d60451d9756eca1f1edcc84248e4f559828 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/StandardUnionObjectInspector.java
 7b2868233f127899c7dca07d4f899b24ae2cbc1b 


Diff: https://reviews.apache.org/r/62321/diff/1/


Testing
---

Added qtest.


Thanks,

Anthony Hsu

[jira] [Created] (HIVE-17530) ClassCastException when converting uniontype

[Anthony Hsu (JIRA)]

Anthony Hsu created HIVE-17530:
--

 Summary: ClassCastException when converting uniontype
 Key: HIVE-17530
 URL: https://issues.apache.org/jira/browse/HIVE-17530
 Project: Hive
  Issue Type: Bug
Affects Versions: 1.1.0, 3.0.0
Reporter: Anthony Hsu
Assignee: Anthony Hsu


To repro:
{noformat}
SET hive.exec.schema.evolution = false;

CREATE TABLE avro_orc_partitioned_uniontype (a uniontype) 
PARTITIONED BY (b int) STORED AS ORC;

INSERT INTO avro_orc_partitioned_uniontype PARTITION (b=1) SELECT 
create_union(1, true, value) FROM src LIMIT 5;

ALTER TABLE avro_orc_partitioned_uniontype SET FILEFORMAT AVRO;

SELECT * FROM avro_orc_partitioned_uniontype;
{noformat}

The exception you get is:
{code}
java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: 
java.lang.ClassCastException: java.util.ArrayList cannot be cast to 
org.apache.hadoop.hive.serde2.objectinspector.UnionObject
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Review Request 62320: HIVE-17529

[Deepak Jaiswal]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62320/
---

Review request for hive, Gopal V and Jason Dere.


Bugs: HIVE-17529
https://issues.apache.org/jira/browse/HIVE-17529


Repository: hive-git


Description
---

Bucket Map Join : Sets incorrect edge type causing execution failure


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorUtils.java e79d1005ec 
  ql/src/test/queries/clientpositive/bucket_map_join_tez1.q aeb244ab8a 
  ql/src/test/queries/clientpositive/bucket_map_join_tez2.q 37989ecc9d 
  ql/src/test/results/clientpositive/llap/bucket_map_join_tez1.q.out f76356658d 
  ql/src/test/results/clientpositive/llap/bucket_map_join_tez2.q.out 57c792f654 


Diff: https://reviews.apache.org/r/62320/diff/1/


Testing
---


Thanks,

Deepak Jaiswal

[jira] [Created] (HIVE-17529) Bucket Map Join : Sets incorrect edge type causing execution failure

[Deepak Jaiswal (JIRA)]

Deepak Jaiswal created HIVE-17529:
-

 Summary: Bucket Map Join : Sets incorrect edge type causing 
execution failure
 Key: HIVE-17529
 URL: https://issues.apache.org/jira/browse/HIVE-17529
 Project: Hive
  Issue Type: Bug
Reporter: Deepak Jaiswal
Assignee: Deepak Jaiswal


If while traversing the tree to generate tasks, a bucket mapjoin may set its 
edge as CUSTOM_SIMPLE_EDGE against CUSTOM_EDGE if the bigtable is already not 
traversed causing Tez to assert and fail the vertex.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 62309: HIVE-17465 Fix filters cardinality for AND predicates

[Vineet Garg]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62309/
---

(Updated Sept. 14, 2017, 12:02 a.m.)


Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-17465
https://issues.apache.org/jira/browse/HIVE-17465


Repository: hive-git


Description
---

This patch fixes estimation of cardinality for filter with AND predicates.


Diffs (updated)
-

  data/files/filterCard.txt PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
 423913b56b 
  ql/src/test/results/clientpositive/annotate_stats_deep_filters.q.out 
6e2975e671 
  ql/src/test/results/clientpositive/annotate_stats_join_pkfk.q.out e04c1c6bc5 
  ql/src/test/results/clientpositive/llap/explainuser_1.q.out 48c02a24ad 
  ql/src/test/results/clientpositive/llap/vector_decimal_cast.q.out c45210e562 
  ql/src/test/results/clientpositive/llap/vector_if_expr.q.out 32d100180d 
  ql/src/test/results/clientpositive/llap/vectorization_0.q.out db76eaaf1b 
  ql/src/test/results/clientpositive/llap/vectorization_10.q.out f06c2dbcc6 
  ql/src/test/results/clientpositive/llap/vectorization_17.q.out 1c07962d83 
  ql/src/test/results/clientpositive/llap/vectorization_7.q.out 6c32ccf42d 
  ql/src/test/results/clientpositive/llap/vectorization_short_regress.q.out 
66764cf42b 
  ql/src/test/results/clientpositive/llap/vectorized_nested_mapjoin.q.out 
2a95065203 
  ql/src/test/results/clientpositive/mapjoin47.q.out 2904b68813 
  ql/src/test/results/clientpositive/perf/query23.q.out dde707d0ea 
  ql/src/test/results/clientpositive/smb_mapjoin_47.q.out 074600ef02 
  ql/src/test/results/clientpositive/spark/vectorization_10.q.out 1f953575e3 
  ql/src/test/results/clientpositive/spark/vectorization_12.q.out 4e26314499 
  ql/src/test/results/clientpositive/spark/vectorization_17.q.out b43c506c6f 
  ql/src/test/results/clientpositive/vectorization_10.q.out d2d9bf644b 
  ql/src/test/results/clientpositive/vectorization_12.q.out 63ea984a33 
  ql/src/test/results/clientpositive/vectorization_17.q.out c0186082df 
  ql/src/test/results/clientpositive/vectorization_7.q.out 218d307915 


Diff: https://reviews.apache.org/r/62309/diff/2/

Changes: https://reviews.apache.org/r/62309/diff/1-2/


Testing
---

* Added new tests
* Updated existing tests


Thanks,

Vineet Garg

Re: Review Request 62309: HIVE-17465 Fix filters cardinality for AND predicates

[Ashutosh Chauhan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62309/#review185348
---




ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
Line 355 (original), 357 (patched)


This comment can be deleted.



ql/src/test/results/clientpositive/filter_cardinality.q.out
Lines 1 (patched)


No need for new tests. Plenty of coverage in existing tests.



ql/src/test/results/clientpositive/join47.q.out
Line 363 (original), 363 (patched)


Is this expected? Seems like patch handled expressions only in Filter not 
in Join.
Also, new estimates are suspiciously too low.



ql/src/test/results/clientpositive/join47.q.out
Line 562 (original), 562 (patched)


Expected ?


- Ashutosh Chauhan


On Sept. 13, 2017, 10:14 p.m., Vineet Garg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62309/
> ---
> 
> (Updated Sept. 13, 2017, 10:14 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-17465
> https://issues.apache.org/jira/browse/HIVE-17465
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This patch fixes estimation of cardinality for filter with AND predicates.
> 
> 
> Diffs
> -
> 
>   data/files/filterCard.txt PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
>  423913b56b 
>   ql/src/test/queries/clientpositive/filter_cardinality.q PRE-CREATION 
>   ql/src/test/results/clientpositive/annotate_stats_deep_filters.q.out 
> 6e2975e671 
>   ql/src/test/results/clientpositive/annotate_stats_join_pkfk.q.out 
> e04c1c6bc5 
>   ql/src/test/results/clientpositive/filter_cardinality.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/join47.q.out 20dce5d3bb 
>   ql/src/test/results/clientpositive/llap/explainuser_1.q.out 48c02a24ad 
>   ql/src/test/results/clientpositive/llap/vector_decimal_cast.q.out 
> c45210e562 
>   ql/src/test/results/clientpositive/llap/vector_if_expr.q.out 32d100180d 
>   ql/src/test/results/clientpositive/llap/vectorization_0.q.out db76eaaf1b 
>   ql/src/test/results/clientpositive/llap/vectorization_10.q.out f06c2dbcc6 
>   ql/src/test/results/clientpositive/llap/vectorization_17.q.out 1c07962d83 
>   ql/src/test/results/clientpositive/llap/vectorization_7.q.out 6c32ccf42d 
>   ql/src/test/results/clientpositive/llap/vectorization_short_regress.q.out 
> 66764cf42b 
>   ql/src/test/results/clientpositive/llap/vectorized_nested_mapjoin.q.out 
> 2a95065203 
>   ql/src/test/results/clientpositive/mapjoin47.q.out 2904b68813 
>   ql/src/test/results/clientpositive/perf/query23.q.out dde707d0ea 
>   ql/src/test/results/clientpositive/smb_mapjoin_47.q.out 074600ef02 
>   ql/src/test/results/clientpositive/spark/vectorization_10.q.out 1f953575e3 
>   ql/src/test/results/clientpositive/spark/vectorization_12.q.out 4e26314499 
>   ql/src/test/results/clientpositive/spark/vectorization_17.q.out b43c506c6f 
>   ql/src/test/results/clientpositive/vectorization_10.q.out d2d9bf644b 
>   ql/src/test/results/clientpositive/vectorization_12.q.out 63ea984a33 
>   ql/src/test/results/clientpositive/vectorization_17.q.out c0186082df 
>   ql/src/test/results/clientpositive/vectorization_7.q.out 218d307915 
> 
> 
> Diff: https://reviews.apache.org/r/62309/diff/1/
> 
> 
> Testing
> ---
> 
> * Added new tests
> * Updated existing tests
> 
> 
> Thanks,
> 
> Vineet Garg
> 
>

Review Request 62310: HIVE-17493 Improve PKFK cardinality estimation in Physical planning

[Vineet Garg]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62310/
---

Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-17493
https://issues.apache.org/jira/browse/HIVE-17493


Repository: hive-git


Description
---

This patch improvies the cardinality estimation for PK-FK relationship.


Diffs
-

  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
 423913b56b 


Diff: https://reviews.apache.org/r/62310/diff/1/


Testing
---


Thanks,

Vineet Garg

Review Request 62309: HIVE-17465 Fix filters cardinality for AND predicates

[Vineet Garg]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62309/
---

Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-17465
https://issues.apache.org/jira/browse/HIVE-17465


Repository: hive-git


Description
---

This patch fixes estimation of cardinality for filter with AND predicates.


Diffs
-

  data/files/filterCard.txt PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
 423913b56b 
  ql/src/test/queries/clientpositive/filter_cardinality.q PRE-CREATION 
  ql/src/test/results/clientpositive/annotate_stats_deep_filters.q.out 
6e2975e671 
  ql/src/test/results/clientpositive/annotate_stats_join_pkfk.q.out e04c1c6bc5 
  ql/src/test/results/clientpositive/filter_cardinality.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/join47.q.out 20dce5d3bb 
  ql/src/test/results/clientpositive/llap/explainuser_1.q.out 48c02a24ad 
  ql/src/test/results/clientpositive/llap/vector_decimal_cast.q.out c45210e562 
  ql/src/test/results/clientpositive/llap/vector_if_expr.q.out 32d100180d 
  ql/src/test/results/clientpositive/llap/vectorization_0.q.out db76eaaf1b 
  ql/src/test/results/clientpositive/llap/vectorization_10.q.out f06c2dbcc6 
  ql/src/test/results/clientpositive/llap/vectorization_17.q.out 1c07962d83 
  ql/src/test/results/clientpositive/llap/vectorization_7.q.out 6c32ccf42d 
  ql/src/test/results/clientpositive/llap/vectorization_short_regress.q.out 
66764cf42b 
  ql/src/test/results/clientpositive/llap/vectorized_nested_mapjoin.q.out 
2a95065203 
  ql/src/test/results/clientpositive/mapjoin47.q.out 2904b68813 
  ql/src/test/results/clientpositive/perf/query23.q.out dde707d0ea 
  ql/src/test/results/clientpositive/smb_mapjoin_47.q.out 074600ef02 
  ql/src/test/results/clientpositive/spark/vectorization_10.q.out 1f953575e3 
  ql/src/test/results/clientpositive/spark/vectorization_12.q.out 4e26314499 
  ql/src/test/results/clientpositive/spark/vectorization_17.q.out b43c506c6f 
  ql/src/test/results/clientpositive/vectorization_10.q.out d2d9bf644b 
  ql/src/test/results/clientpositive/vectorization_12.q.out 63ea984a33 
  ql/src/test/results/clientpositive/vectorization_17.q.out c0186082df 
  ql/src/test/results/clientpositive/vectorization_7.q.out 218d307915 


Diff: https://reviews.apache.org/r/62309/diff/1/


Testing
---

* Added new tests
* Updated existing tests


Thanks,

Vineet Garg

[GitHub] hive pull request #250: HIVE-17527: Support replication for rename/move tabl...

[sankarh]

GitHub user sankarh opened a pull request:

https://github.com/apache/hive/pull/250

HIVE-17527: Support replication for rename/move table across database



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sankarh/hive HIVE-17527

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/250.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #250


commit 223682c15e418b5cfbfecf71575229722f3dca25
Author: Sankar Hariappan 
Date:   2017-09-13T19:03:30Z

HIVE-17527: Support replication for rename/move table across database




---

Re: Hive Custom Inputformat mapred.* vs. mapreduce.*

[Jörn Franke]

Ok mapreduce is the engine, but I am talking about the file format. Even with 
Tez or Spark as an engine the fileformat will always be built on mapped.* 
and/or mapreduce.* 
Aside from hive this is also the case in Spark and Flink to leverage the 
benefits to access different storages (S3, HDFS etc) and file formats 
transparently.

> On 13. Sep 2017, at 20:53, Alan Gates  wrote:
> 
> I’m not aware of any plans in Hive to do any more work that uses Map Reduce
> as the execution engine, so I expect Hive will continue to use mapred.
> 
> Alan.
> 
>> On Wed, Sep 13, 2017 at 4:25 AM, Jörn Franke  wrote:
>> 
>> Dear all,
>> 
>> I have developed several custom input formats (e.g. for the Bitcoin
>> blockchain) including a HiveSerde, which are open source.
>> I plan to develop for my HadoopOffice inputformat also a HiveSerde, but I
>> wonder if I should continue to use mapred.* apis or if i should use
>> mapreduce.*
>> 
>> My inputformats support both APIs, but it seems that Hive is one of the
>> last (please correct me here) to use the mapred.* API
>> 
>> Personally, i have no preference, since I support both.
>> 
>> Thank you.
>> 
>> All the best
>> 

Re: Hive Custom Inputformat mapred.* vs. mapreduce.*

[Alan Gates]

I’m not aware of any plans in Hive to do any more work that uses Map Reduce
as the execution engine, so I expect Hive will continue to use mapred.

Alan.

On Wed, Sep 13, 2017 at 4:25 AM, Jörn Franke  wrote:

> Dear all,
>
> I have developed several custom input formats (e.g. for the Bitcoin
> blockchain) including a HiveSerde, which are open source.
> I plan to develop for my HadoopOffice inputformat also a HiveSerde, but I
> wonder if I should continue to use mapred.* apis or if i should use
> mapreduce.*
>
> My inputformats support both APIs, but it seems that Hive is one of the
> last (please correct me here) to use the mapred.* API
>
> Personally, i have no preference, since I support both.
>
> Thank you.
>
> All the best
>

[jira] [Created] (HIVE-17528) Add more q-tests for Hive-on-Spark with Parquet vectorized reader

[Vihang Karajgaonkar (JIRA)]

Vihang Karajgaonkar created HIVE-17528:
--

 Summary: Add more q-tests for Hive-on-Spark with Parquet 
vectorized reader
 Key: HIVE-17528
 URL: https://issues.apache.org/jira/browse/HIVE-17528
 Project: Hive
  Issue Type: Sub-task
Reporter: Vihang Karajgaonkar
Assignee: Vihang Karajgaonkar


Most of the vectorization related q-tests operate on ORC tables using Tez. It 
would be good to add more coverage on a different combination of engine and 
file-format. We can model existing q-tests using parquet tables and run it 
using TestSparkCliDriver



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17527) Support replication for rename/move table across database

[Sankar Hariappan (JIRA)]

Sankar Hariappan created HIVE-17527:
---

 Summary: Support replication for rename/move table across database
 Key: HIVE-17527
 URL: https://issues.apache.org/jira/browse/HIVE-17527
 Project: Hive
  Issue Type: Sub-task
Affects Versions: 2.1.0
Reporter: Sankar Hariappan
Assignee: Sankar Hariappan
 Fix For: 3.0.0


Rename/move table across database should be supported for replication. The 
scenario is as follows.

1. Create 2 databases (db1 and db2) in source cluster.
2. Create the table db1.tbl1.
3. Run bootstrap replication for db1 and db2 to target cluster.
4. Rename db1.tbl1 to db2.tbl1 in source.
5. Run incremental replication for both db1 and db2.
- db1 dump fails telling rename across databases is not supported.
- db2 dump missed the table as no event is generated when moved to db2. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] hive pull request #248: HIVE-17494: Bootstrap REPL DUMP throws exception if ...

[sankarh]

Github user sankarh closed the pull request at:

https://github.com/apache/hive/pull/248


---

Re: Review Request 62228: HIVE-17495: CachedStore: prewarm improvements, refactoring and caching some aggregate stats

[Ashutosh Chauhan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62228/#review185297
---



Also, I don't see any logic for cache invalidation in case partition is added 
or deleted. Did I miss that?

- Ashutosh Chauhan


On Sept. 11, 2017, 9:25 p.m., Vaibhav Gumashta wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62228/
> ---
> 
> (Updated Sept. 11, 2017, 9:25 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Thejas Nair.
> 
> 
> Bugs: HIVE-17495
> https://issues.apache.org/jira/browse/HIVE-17495
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/HIVE-17495
> 
> 
> Diffs
> -
> 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  8d861e4 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 
> dc1245e 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
> bbe13fd 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
> 3053dcb 
>   metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 71982a0 
>   metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 
> 3ba81ce 
>   metastore/src/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java 
> 80b17e0 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/BinaryColumnStatsAggregator.java
>  e6c836b 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/BooleanColumnStatsAggregator.java
>  a34bc9f 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/ColumnStatsAggregator.java
>  a52e5e5 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/ColumnStatsAggregatorFactory.java
>  dfae708 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/DateColumnStatsAggregator.java
>  ee95396 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/DecimalColumnStatsAggregator.java
>  284c12c 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/DoubleColumnStatsAggregator.java
>  bb4a725 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/LongColumnStatsAggregator.java
>  5b1145e 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/StringColumnStatsAggregator.java
>  1b29f92 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
>  4db203d 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
>  fb16cfc 
> 
> 
> Diff: https://reviews.apache.org/r/62228/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Vaibhav Gumashta
> 
>

[jira] [Created] (HIVE-17526) Disable conversion to ACID if table has _copy_N files on branch-1

[Daniel Voros (JIRA)]

Daniel Voros created HIVE-17526:
---

 Summary: Disable conversion to ACID if table has _copy_N files on 
branch-1
 Key: HIVE-17526
 URL: https://issues.apache.org/jira/browse/HIVE-17526
 Project: Hive
  Issue Type: Bug
Reporter: Daniel Voros
Assignee: Daniel Voros
 Fix For: 1.3.0


As discussed in HIVE-16177, non-ACID to ACID conversion can lead to data loss 
if the table has *_copy_N files.

The patch for HIVE-16177 is quite massive and would basically need a 
reimplementation to apply for branch-1 since the related code paths have 
diverged a lot. We could disable the conversion to ACID if there are *_copy_N 
files instead.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Hive Custom Inputformat mapred.* vs. mapreduce.*

[Jörn Franke]

Dear all,

I have developed several custom input formats (e.g. for the Bitcoin
blockchain) including a HiveSerde, which are open source.
I plan to develop for my HadoopOffice inputformat also a HiveSerde, but I
wonder if I should continue to use mapred.* apis or if i should use
mapreduce.*

My inputformats support both APIs, but it seems that Hive is one of the
last (please correct me here) to use the mapred.* API

Personally, i have no preference, since I support both.

Thank you.

All the best

Review Request 62280: HIVE-16827: Merge stats task and column stats task into a single task

[Zoltan Haindrich]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62280/
---

Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-16827
https://issues.apache.org/jira/browse/HIVE-16827


Repository: hive-git


Description
---

this was originally part of HIVE-13567


Diffs
-

  common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java 
7c27d07024e4d6f21e3b1d24a5efcc1325e64d6e 
  common/src/java/org/apache/hadoop/hive/common/jsonexplain/Vertex.java 
b7dc88c93984a37a5df7ec8258c3e1e375cf7878 
  itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestMTQueries.java 
ad2baa2e265d2d0ffb94859e5141e4b38f2909b5 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 
4a9af80fdc88e267b9aef97adcbe9e6aa18196dc 
  itests/src/test/resources/testconfiguration.properties 
772113acdac100ac87db66eeecbbd9df10f184fb 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 
28c3cfed6d74a0788fce889adb7de22db42f5c13 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
df01b2578c4fb3ab4f99f7f7b0c921bdb2bc 
  metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
bbe13fd77b5a0cfa7a6818f7a9e23985772acf3d 
  metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
0a80241b77171421e22acfaf34d71ff20d7c17f5 
  
metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/merge/ColumnStatsMergerFactory.java
 66be52413956b261373b2132c5678a204662c79e 
  ql/src/java/org/apache/hadoop/hive/ql/DriverContext.java 
f43992c85d4695f6278eaa36420ef5ba331f5200 
  ql/src/java/org/apache/hadoop/hive/ql/exec/BasicStatsTask.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java 
2b2c004fea84838a438d1404337ca60fd300664d 
  ql/src/java/org/apache/hadoop/hive/ql/exec/StatsNoJobTask.java 
9c3a664b9aea2d6e050ffe2d7626127827dbc52a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java 
c22d69bb19064fe363276478ec89dd08db5a8705 
  ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java 
91ac4bf985777599afa392594c5e2e95691caacd 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 
d661f10c407776a9123d074b3cf3dbcb1d5f0508 
  
ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java 
61f6a7c4ff38447db0ac2610e7308f4e710580ab 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 
9297a0b87454bf37b1a4c68327407cada6b37232 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
da153e36d2d0a4e0de1a68e8f26ead963a2317a6 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/MemoryDecider.java 
3a20cfe7ac693340bda97c345d1603d312dbafa3 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SerializeFilter.java 
dc433fed2230caa0afbb270c2e05fa8f356709cf 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
a054abb127d5a67c845647e9d9c4f3c174791750 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ExplainSemanticAnalyzer.java 
7a0d4a752e6dfd02575f368168a4091de29aebf4 
  ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java 
1b0a2f066161da3bf912f24c55aa0a0c4ccf878d 
  ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 
fa79700df71c116f229bb9cd25a4ed61f3a38bb0 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ProcessAnalyzeTable.java 
b6d7ee8a92d5c721995221cf50554f4ea6974edf 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
bc6e0d586e70fc7bf9b5bac231abaa7a14609069 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java 
08a8f00e06afc756282f42e30929fd31afb5 
  
ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkProcessAnalyzeTable.java 
a2876e1d4f35ce9d0114fbec73cc644d68dade57 
  ql/src/java/org/apache/hadoop/hive/ql/plan/BasicStatsWork.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsDesc.java 
97f323f4b7e89677fa2037f87df8735fc59d5b21 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsWork.java 
76811b1a93f255dddc154d468be4bead1a254e60 
  ql/src/java/org/apache/hadoop/hive/ql/plan/StatsNoJobWork.java 
77c04f6c6e5959e8ed5d891075912692a2f2ecfe 
  ql/src/java/org/apache/hadoop/hive/ql/plan/StatsWork.java 
a5050c5368d041c61694c0734c03d6577bbf85a8 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands.java 
bff9884aa1b10875d33b9a0b90c164fae144efef 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java 
0e0fca313ea54fb684d197b91b73e28f0e26ae39 
  ql/src/test/org/apache/hadoop/hive/ql/TxnCommandsBaseForTests.java 
PRE-CREATION 
  ql/src/test/queries/clientpositive/autoColumnStats_1.q 
7955b0723356f74169f58c0387e4e268486a04de 
  ql/src/test/queries/clientpositive/autoColumnStats_10.q PRE-CREATION 
  ql/src/test/queries/clientpositive/bucket_map_join_tez2.q 
37989ecc9d14d38d1633e5dd415ae2ceaf8028ed 
  ql/src/test/queries/clientpositive/bucket_num_reducers.q