[Tim]
>> At the start, obmalloc never returned arenas to the system. The vast
>> majority of users were fine with that.
[Neil]
> Yeah, I was totally fine with that back in the day. However, I
> wonder now if there is a stronger reason to try to free memory back
> to the OS. Years ago, people would typically have swap space that
> was as large or larger than their real RAM. So, if the OS had to
> swap out unused pages, it wasn't a big deal. Now that disks are
> relatively so much slower and RAM is larger, people don't have as
> much swap. Some Linux systems get setup without any. Freeing
> arenas seems more important than it used to be.
See my earlier "right thing to do ... anticipating" ;-) It was clear
enough then that processor speeds were going to continue to increase
much faster than disk speeds, although the _primary_ driver at the
time was the then-small but increasing number of long-running Python
programs running big (for the time) data-crunching problems that
repeatedly went through stages of low and high memory need.
> OTOH, I don't think obmalloc should try too hard. The whole point of
> the small object allocator is to be really fast. Anti-fragmentation
> heuristics are going to slow it down. As far as I'm concerned, it
> works well enough as it is.
Freeing arenas _already_ slowed it down. All the cruft to keep arenas
sorted by number of free pools was far from free. Before that, free
pools for a given size class were singly-linked together directly
(regardless of which arenas they were in), and we just used the head
of that list. Arenas weren't involved at all. Indeed, pools didn't
even point back to their arenas.
So major code and data changes were needed to implement the "take from
the non-full pool in the most heavily populated arena" heuristic. The
other heuristic worth considering is "take from the non-full pool with
the smallest arena address", which directly strives to let the
higher-addressed arenas free up.
Since they're both just poke-&-hope, it's hard to out-think them. But
it's possible one would consistently work better than the other, and
that even _which_ one could change depending on arena and pool sizes.
And should note that "poke-&-hope" is the best that can be hoped for,
since it's impossible to predict the lifetime of an object when space
for it is requested, and we can't move the object after its address is
returned to the caller.
More amusing: address_in_range() didn't exist either at first.
Instead a pool header contained its own address and a "magic"
constant. So, e.g.,
if (pool->pooladdr != pool || pool->magic != POOL_MAGIC) {
// This ain't ours! Pass it to the system malloc family instead.
}
That was unlikely to believe something that _had_ come from the system
malloc really belonged to obmalloc, but proved very easy to trick into
making that error, even from pure Python code.
Good times ;-)
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/GLE6UC7EKGZLBTFJVHUMMCKYXRRWU5GB/
