[
https://issues.apache.org/jira/browse/HIVE-22275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16944167#comment-16944167
]
Prasanth Jayachandran commented on HIVE-22275:
----------------------------------------------
lgtm, +1. Looks like when the cleanup happens the queryId from session conf is
retrieved which will only reflect the last query id and ends up leaking all the
query ids before it.
> OperationManager.queryIdOperation does not properly clean up multiple queryIds
> ------------------------------------------------------------------------------
>
> Key: HIVE-22275
> URL: https://issues.apache.org/jira/browse/HIVE-22275
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
> Reporter: Jason Dere
> Assignee: Jason Dere
> Priority: Major
> Attachments: HIVE-22275.1.patch, HIVE-22275.2.patch
>
>
> In the case that multiple statements are run by a single Session before being
> cleaned up, it appears that OperationManager.queryIdOperation is not cleaned
> up properly.
> See the log statements below - with the exception of the first "Removed
> queryId:" log line, the queryId listed during cleanup is the same, when each
> of these handles should have their own queryId. Looks like only the last
> queryId executed is being cleaned up.
> As a result, HS2 can run out of memory as OperationManager.queryIdOperation
> grows and never cleans these queryIds/Operations up.
> {noformat}
> 2019-09-13T08:37:36,785 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=dfed4c18-a284-4640-9f4a-1a20527105f9]
> 2019-09-13T08:37:38,432 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Removed queryId: hive_20190913083736_c49cf3cc-cfe8-48a1-bd22-8b924dfb0396
> corresponding to operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=dfed4c18-a284-4640-9f4a-1a20527105f9] with tag: null
> 2019-09-13T08:37:38,469 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=24d0030c-0e49-45fb-a918-2276f0941cfb]
> 2019-09-13T08:37:52,662 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=b983802c-1dec-4fa0-8680-d05ab555321b]
> 2019-09-13T08:37:56,239 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=75dbc531-2964-47b2-84d7-85b59f88999c]
> 2019-09-13T08:38:02,551 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=72c79076-9d67-4894-a526-c233fa5450b2]
> 2019-09-13T08:38:10,558 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=17b30a62-612d-4b70-9ba7-4287d2d9229b]
> 2019-09-13T08:38:16,930 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=ea97e99d-cc77-470b-b49a-b869c73a4615]
> 2019-09-13T08:38:20,440 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=a277b789-ebb8-4925-878f-6728d3e8c5fb]
> 2019-09-13T08:38:26,303 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=9a023ab8-aa80-45db-af88-94790cc83033]
> 2019-09-13T08:38:30,791 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=b697c801-7da0-4544-bcfa-442eb1d3bd77]
> 2019-09-13T08:39:10,187 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Adding operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=bda93c8f-0822-4592-a61c-4701720a1a5c]
> 2019-09-13T08:39:15,471 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Removed queryId: hive_20190913083910_c4809ca8-d8db-423c-8b6d-fbe3eee89971
> corresponding to operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=24d0030c-0e49-45fb-a918-2276f0941cfb] with tag: null
> 2019-09-13T08:39:15,507 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Removed queryId: hive_20190913083910_c4809ca8-d8db-423c-8b6d-fbe3eee89971
> corresponding to operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=b983802c-1dec-4fa0-8680-d05ab555321b] with tag: null
> 2019-09-13T08:39:15,538 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Removed queryId: hive_20190913083910_c4809ca8-d8db-423c-8b6d-fbe3eee89971
> corresponding to operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=75dbc531-2964-47b2-84d7-85b59f88999c] with tag: null
> 2019-09-13T08:39:15,572 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Removed queryId: hive_20190913083910_c4809ca8-d8db-423c-8b6d-fbe3eee89971
> corresponding to operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=72c79076-9d67-4894-a526-c233fa5450b2] with tag: null
> 2019-09-13T08:39:15,600 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Removed queryId: hive_20190913083910_c4809ca8-d8db-423c-8b6d-fbe3eee89971
> corresponding to operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=17b30a62-612d-4b70-9ba7-4287d2d9229b] with tag: null
> 2019-09-13T08:39:15,629 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Removed queryId: hive_20190913083910_c4809ca8-d8db-423c-8b6d-fbe3eee89971
> corresponding to operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=ea97e99d-cc77-470b-b49a-b869c73a4615] with tag: null
> 2019-09-13T08:39:15,657 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Removed queryId: hive_20190913083910_c4809ca8-d8db-423c-8b6d-fbe3eee89971
> corresponding to operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=a277b789-ebb8-4925-878f-6728d3e8c5fb] with tag: null
> 2019-09-13T08:39:15,692 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Removed queryId: hive_20190913083910_c4809ca8-d8db-423c-8b6d-fbe3eee89971
> corresponding to operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=9a023ab8-aa80-45db-af88-94790cc83033] with tag: null
> 2019-09-13T08:39:15,723 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Removed queryId: hive_20190913083910_c4809ca8-d8db-423c-8b6d-fbe3eee89971
> corresponding to operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=b697c801-7da0-4544-bcfa-442eb1d3bd77] with tag: null
> 2019-09-13T08:39:15,754 INFO [8eaa1601-f045-4ad5-9c2e-1e5944b75f6a
> HiveServer2-Handler-Pool: Thread-202]: operation.OperationManager (:()) -
> Removed queryId: hive_20190913083910_c4809ca8-d8db-423c-8b6d-fbe3eee89971
> corresponding to operation: OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=bda93c8f-0822-4592-a61c-4701720a1a5c] with tag: null
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
