[
https://issues.apache.org/jira/browse/ARROW-16697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17550353#comment-17550353
]
Lubo Slivka commented on ARROW-16697:
-------------------------------------
Hi,
I have spent some time today fiddling about with the glibc tunables and malloc
and things become much brighter.
As I understand it, malloc by default tends to create per-thread arenas; the
limit on number of arenas is: "For 32-bit systems the limit is twice the number
of cores online and on 64-bit systems, it is 8 times the number of cores
online."
This imho directly answers the 'high watermark' we contemplated above. More
concurrent clients -> more threads on the server -> more arenas.
The *glibc.malloc.arena_max* tunable (or MALLOC_ARENA_MAX env var) can help to
keep the high watermark low at the cost of contention during allocations.
---
However, what I find is that the limiting the number of arenas is not really
needed. What seems important is the *glibc.malloc.trim_threshold:* "If this
tunable is not set, the default value is set as 128 KB and the threshold is
adjusted dynamically to suit the allocation patterns of the program.".
While sounding reasonable, this default behavior does not seem to produce
expected results (even if the record batches that go over wire are 1MB large).
I'm sure there is perfectly valid explanation behind it all but alas, I cannot
spend more time digging into it now.
Setting this threshold explicitly helps keeping the RSS down during DoPuts and
after they complete brings the RSS down very close to the starting point.
So for instance something like this goes a long way:
{code:java}
export GLIBC_TUNABLES=glibc.malloc.trim_threshold=524288 {code}
---
I think this trim tuning, perhaps in combination with explicit trim triggered
every once in a while is what I'll go with.
Thanks [~lidavidm] for helping out and setting me on the right track. Shall I
close this ticket then?
--L
> [FlightRPC][Python] Server seems to leak memory during DoPut
> ------------------------------------------------------------
>
> Key: ARROW-16697
> URL: https://issues.apache.org/jira/browse/ARROW-16697
> Project: Apache Arrow
> Issue Type: Bug
> Reporter: Lubo Slivka
> Assignee: David Li
> Priority: Major
> Attachments: leak_repro_client.py, leak_repro_server.py, massif.txt,
> massif_client.txt, sample.csv.gz
>
>
> Hello,
> We are stress testing our Flight RPC server (PyArrow 8.0.0) with write-heavy
> workloads and are running into what appear to be memory leaks.
> The server is under pressure by a number of separate clients doing DoPut.
> What we are seeing is that server's memory usage only ever goes up until the
> server finally gets whacked by k8s due to hitting memory limit.
> I have spent many hours fishing through our code for memory leaks with no
> success. Even short-circuiting all our custom DoPut handling logic does not
> alleviate the situation. This led me to create a reproducer that uses nothing
> but PyArrow and I see the server process memory only increasing similar to
> what we see on our servers.
> The reproducer is in attachments + I included the test CSV file (20MB) that I
> use for my tests. Few notes:
> * The client code has multiple threads, each emulating a separate Flight
> Client
> * There are two variants where I see slightly different memory usage
> characteristic:
> ** _do_put_with_client_reuse << one client opened at start of thread, then
> hammering many puts, finally closing the client; leaks appear to happen
> faster in this variant
> ** _do_put_with_client_per_request << client opens & connects, does put,
> then disconnects; loop like this many times; leaks appear to happen slower in
> this variant if there are less concurrent clients; increasing number of
> threads 'helps'
> * The server code handling do_put reads batch-by-batch & does nothing with
> the chunks
> Also one interesting (but highly likely unrelated thing) that I keep noticing
> is that _sometimes_ FlightClient takes long time to close (like 5seconds). It
> happens intermittently.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
