[
https://issues.apache.org/jira/browse/IMPALA-14763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18075726#comment-18075726
]
ASF subversion and git services commented on IMPALA-14763:
----------------------------------------------------------
Commit 1a73414d729e4ab8e519444aac7bfd6b3019e60a in impala's branch
refs/heads/master from Yida Wu
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=1a73414d7 ]
IMPALA-14763: Prevent admissiond OOM during request decompression
When admissiond is close to its memory limit and a very large
queued query is dequeued, decompression of this compressed
request can push memory usage over the limit and cause an OOM.
Previously in IMPALA-14493, memory checks TCMalloc's BYTES_IN_USE
to provide memory safeguard on Submission for uncompressed
requests, but after IMPALA-14661, we need to consider the
decompression cases. This patch adds memory safeguard for
compressed requests, mainly the decompression will happen on
Submission or Dequeue.
We put all the rejection logic into a static function
RejectForAdmissionServiceMemory(), and introduce a new memory
tracker, pending_decompression_mem_tracker, to track the total
uncompressed size of pending compressed requests.
RejectForAdmissionServiceMemory() compares the current
tcmalloc bytes-in-use plus the additional memory to reserve
against the process memory limit.
For compressed requests, we first add the request’s uncompressed
size to pending_decompression_mem_tracker, then pass the total
pending uncompressed size as the additional reserved memory to
RejectForAdmissionServiceMemory(), ensuring thread safety.
For uncompressed requests, the additional memory is zero.
If the check fails, RejectForAdmissionServiceMemory() returns
an error and admissiond rejects the query.
Additionally, to prevent early decompression for queued
compressed requests when GetQueryStatus() is called, we now
cache the TQueryOptions inside AdmissionExecRequestCompressed
during the first decompression. WaitOnQueued() uses these
cached options for the AC_AFTER_ADMISSION_OUTCOME debug
action instead of redecompressing the whole request.
Testing:
Added a new test to check compressed requests being rejected
on Submission. Manually verified that the safeguard also works
at Dequeue, an automated test for the Dequeue case was a bit
flaky to include.
Passed exhaustive test test_admission_controller.py.
Change-Id: I196455f445f0644d89467a23b4ec1f64f184f2db
Reviewed-on: http://gerrit.cloudera.org:8080/24055
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Add memory guard before decompressing queries in admissiond
> -----------------------------------------------------------
>
> Key: IMPALA-14763
> URL: https://issues.apache.org/jira/browse/IMPALA-14763
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Reporter: Yida Wu
> Assignee: Yida Wu
> Priority: Critical
> Fix For: Impala 5.0.0
>
>
> When admissiond is close to its memory limit and a very large query is
> dequeued, decompression (when compression is enabled) can push memory usage
> over the limit and cause OOM.
> We should add a protection similar to IMPALA-14493 to reject the queued query
> before decompression if current_memory + uncompressed_size > memory_limit.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
