[
https://issues.apache.org/jira/browse/HIVE-16334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15954920#comment-15954920
]
Peter Vary commented on HIVE-16334:
-----------------------------------
[~stakiar]: I considered the following things:
- The query is stored in HS2, but the locks should be displayed on every client
even when HS2 is in HA. So if we want to keep the query string in a shared
database, or another ZooKeeper node next to the lock.
- Since the query string is read by humans, I do not think that it is worth to
display a string which is longer than 1000 chars.
ZooKeeper scales fairly well if we limit the size of the nodes, so considering
the points above I think it might not worth the complexity to externalize the
query string from the locks.
What do you think? Did I miss some important point?
Thanks for the review!
Peter
> Query lock contains the query string, which can cause OOM on ZooKeeper
> ----------------------------------------------------------------------
>
> Key: HIVE-16334
> URL: https://issues.apache.org/jira/browse/HIVE-16334
> Project: Hive
> Issue Type: Improvement
> Components: Locking
> Reporter: Peter Vary
> Assignee: Peter Vary
> Attachments: HIVE-16334.patch
>
>
> When there are big number of partitions in a query this will result in a huge
> number of locks on ZooKeeper. Since the query object contains the whole query
> string this might cause serious memory pressure on the ZooKeeper services.
> It would be good to have the possibility to truncate the query strings that
> are written into the locks
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
