Yida Wu created IMPALA-14276:
--------------------------------
Summary: Suspected memory leak when requests are rejected in
standalone admissiond
Key: IMPALA-14276
URL: https://issues.apache.org/jira/browse/IMPALA-14276
Project: IMPALA
Issue Type: Bug
Components: Backend
Reporter: Yida Wu
Assignee: Yida Wu
During testing, I observed that memory usage (tcmalloc.bytes-in-use) keeps
increasing when requests are rejected or time out via the admissiond service.
This behavior seems to occur only when admissiond is running as a standalone
process. I did not observe the same pattern when admissiond is embedded within
the impalad process.
Repro:
1. Start the cluster with standalone admissiond:
{code:java}
$IMPALA_HOME/bin/start-impala-cluster.py
--admissiond_args='--max_admission_queue_size=1000000
--default_pool_max_queued=100000 --default_pool_max_requests=1'
--impalad_args='--fe_service_threads=1024' --num_coordinators=2
--enable_admission_service
{code}
2. Run one long running query to hold the slot:
{code:java}
select sleep(150000000);
{code}
3. Run the script to keep running queries, and they will time out:
{code:java}
#!/bin/bash
MAX_RUNS=10
for ((j = 1; j <= MAX_RUNS; j++)); do
for i in {1..500}; do
nohup $IMPALA_HOME/bin/impala-shell.sh -f test_sleep.sql > "out.log" 2>&1 &
sleep 0.2
done
disown -a
sleep 60
done
{code}
4. Monitor memory usage. A steady increase over time like below. Given that all
incoming requests are being rejected, we wouldn’t expect memory to grow like
this, which suggests a possible memory leak.
{code:java}
tcmalloc.bytes-in-use 115.51 MB -> 142.04 MB -> 168.49 MB -> 225.71 MB
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
