[
https://issues.apache.org/jira/browse/HIVE-21391?focusedWorklogId=261227&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-261227
]
ASF GitHub Bot logged work on HIVE-21391:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 17/Jun/19 08:24
Start Date: 17/Jun/19 08:24
Worklog Time Spent: 10m
Work Description: prasanthj commented on issue #641: HIVE-21391: Fix
VRBsQueue size estimate
URL: https://github.com/apache/hive/pull/641#issuecomment-502584489
+1
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 261227)
Time Spent: 20m (was: 10m)
> LLAP: Pool of column vector buffers can cause memory pressure
> -------------------------------------------------------------
>
> Key: HIVE-21391
> URL: https://issues.apache.org/jira/browse/HIVE-21391
> Project: Hive
> Issue Type: Bug
> Components: llap
> Affects Versions: 4.0.0, 3.2.0
> Reporter: Prasanth Jayachandran
> Assignee: Prasanth Jayachandran
> Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21391.1.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Where there are too many columns (in the order of 100s), with decimal, string
> types the column vector pool of buffers created here
> [https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/EncodedDataConsumer.java#L59]
> can cause memory pressure.
> Example:
> 128 (poolSize) * 300 (numCols) * 1024 (batchSize) * 80 (decimalSize) ~= 3GB
> The pool size keeps increasing when there is slow consumer but fast llap io
> (SSDs) leading to GC pressure when all LLAP io threads read splits from same
> table.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
