[
https://issues.apache.org/jira/browse/HIVE-21509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Szita updated HIVE-21509:
------------------------------
Status: In Progress (was: Patch Available)
> LLAP may cache corrupted column vectors and return wrong query result
> ---------------------------------------------------------------------
>
> Key: HIVE-21509
> URL: https://issues.apache.org/jira/browse/HIVE-21509
> Project: Hive
> Issue Type: Bug
> Components: llap
> Reporter: Adam Szita
> Assignee: Adam Szita
> Priority: Major
> Attachments: HIVE-21509.0.wip.patch, HIVE-21509.1.wip.patch,
> HIVE-21509.2.patch, HIVE-21509.3.patch, HIVE-21509.4.patch
>
>
> In some scenarios, LLAP might store column vectors in cache that are getting
> reused and reset just before their original content would be written.
> The issue is a concurrency issue and is thereby flaky. It is not easy to
> reproduce, but the odds of surfacing this issue can by improved by setting
> LLAP executor and IO thread counts this way:
> * set hive.llap.daemon.num.executors=32;
> * set hive.llap.io.threadpool.size=1;
> * using TPCDS input data of store_sales table, have at least a couple of
> 100k's of rows, and use text format:
> {code:java}
> ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
> WITH SERDEPROPERTIES ( 'field.delim'='|', 'serialization.format'='|')
> STORED AS INPUTFORMAT 'org.apache.hadoop.mapred.TextInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'{code}
> * having more splits increases the issue showing itself, so it is worth to
> _set tez.grouping.min-size=1024; set tez.grouping.max-size=1024;_
> * run query on this this table: select min(ss_sold_date_sk) from store_sales;
> The first query result is correct (2450816 in my case). Repeating the query
> will trigger reading from LLAP cache and produce a wrong result: 0.
> If one wants to make sure of running into this issue, place a
> Thread.sleep(250) at the beginning of VectorDeserializeOrcWriter#run().
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
