[Python-Dev] Re: radix tree arena map for obmalloc

[Antoine Pitrou]

On Mon, 17 Jun 2019 11:15:29 +0900
Inada Naoki  wrote:
> 
> Increasing pool size is one obvious way to fix these problems.
> I think 16KiB pool size and 2MiB (huge page size of x86) arena size is
> a sweet spot for recent web servers (typically, about 32 threads, and
> 64GiB), but there is no evidence about it.

Note that the OS won't give a huge page automatically, because memory
management becomes much more inflexible then.

For example, the Linux madvise() man page has this to say about
MADV_HUGEPAGE:

  This feature is primarily aimed at  applications  that
  use large mappings of data and access large regions of
  that memory at a time  (e.g.,  virtualization  systems
  such as QEMU).  It can very easily waste memory (e.g.,
  a 2 MB mapping that only ever  accesses  1  byte  will
  result  in  2 MB  of  wired memory instead of one 4 KB
  page).  See the Linux kernel  source  file  Documenta‐
  tion/vm/transhuge.txt for more details.

I'm not sure a small objects allocator falls into the right use case
for huge pages.

Regards

Antoine.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GDRMLWAI26TSRMX6HNJ6C5X7RRJL/

[Python-Dev] Re: radix tree arena map for obmalloc

[Inada Naoki]

On Mon, Jun 17, 2019 at 5:18 PM Antoine Pitrou  wrote:
>
> On Mon, 17 Jun 2019 11:15:29 +0900
> Inada Naoki  wrote:
> >
> > Increasing pool size is one obvious way to fix these problems.
> > I think 16KiB pool size and 2MiB (huge page size of x86) arena size is
> > a sweet spot for recent web servers (typically, about 32 threads, and
> > 64GiB), but there is no evidence about it.
>
> Note that the OS won't give a huge page automatically, because memory
> management becomes much more inflexible then.

When we used contiguous 2MB pages, Linux may use Huge Page
implicitly when Transparent Huge Page is enabled.

Then, if we munmap one page of the 2MB, the kernel will split
the huge page into small many pages again.
I don't know it really happens on Python applications,
but using 2MB arena will reduce the risk of performance penalty
of page splitting.

In web applications, it's common to one Python worker process
use 50~100+ MiB RSS.  2MB arena seems reasonable for those
applications.

I am not proposing making this default.
I just meant this may be a reasonable configuration for many Python
users who create medium~large size web apps.

Regards,

-- 
Inada Naoki  
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VGXTBA6JBYOMXVTFVZ7NEINLPBQ4ZYFT/

[Python-Dev] Re: radix tree arena map for obmalloc

[Antoine Pitrou]

Le 17/06/2019 à 10:55, Inada Naoki a écrit :
> On Mon, Jun 17, 2019 at 5:18 PM Antoine Pitrou  wrote:
>>
>> On Mon, 17 Jun 2019 11:15:29 +0900
>> Inada Naoki  wrote:
>>>
>>> Increasing pool size is one obvious way to fix these problems.
>>> I think 16KiB pool size and 2MiB (huge page size of x86) arena size is
>>> a sweet spot for recent web servers (typically, about 32 threads, and
>>> 64GiB), but there is no evidence about it.
>>
>> Note that the OS won't give a huge page automatically, because memory
>> management becomes much more inflexible then.
> 
> When we used contiguous 2MB pages, Linux may use Huge Page
> implicitly when Transparent Huge Page is enabled.

But it's not enabled by default...  And there are reasons for that (see
the manpage I quoted).

> In web applications, it's common to one Python worker process
> use 50~100+ MiB RSS.  2MB arena seems reasonable for those
> applications.

Perhaps, but what is the problem you are trying to solve?  Do you have
evidence that memory management of those 50-100 MB is costly?

Perhaps those 50-100 MB are allocated at worker startup (module
initialization structures, docstrings...) and only get deallocated at
the end, so they aren't really a problem for arena allocation cost.

Regards

Antoine.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/Z2AO6AFBCYDJK7HM3NZ6SCTWXSUZ36TI/

[Python-Dev] Re: radix tree arena map for obmalloc

[Inada Naoki]

On Mon, Jun 17, 2019 at 6:14 PM Antoine Pitrou  wrote:
> But it's not enabled by default...  And there are reasons for that (see
> the manpage I quoted).

Uh, then if people want to use huge page, they need to enable it on system wide,
or add madvice in obmalloc.c.

> > In web applications, it's common to one Python worker process
> > use 50~100+ MiB RSS.  2MB arena seems reasonable for those
> > applications.
>
> Perhaps, but what is the problem you are trying to solve?  Do you have
> evidence that memory management of those 50-100 MB is costly?
>

I just meant we may be able to utilize THP if we provide large arena option.
In other words, *if* we provide configure option to increase arena & pool size,
2MB arena seems reasonable to me.  That's all I wanted to say here.

I didn't mean utilizing THP is the main motivation to increase arena size.
People who want to use huge page may have a problem to solve by huge page.
But I don't have it.

Regards,
-- 
Inada Naoki  
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/S3GELVBIRQS5LOA72OIPPUNPNZGEUV2J/

[Python-Dev] How to (best) organize platform dependent blocks of Python code

[Michael]

CONCERN: How to organize platform dependent blocks of code
POLICY/RECOMMENDATION: (today) seems to be: arbitrary, although
sys.platform, seems to be the favorite.

* I, as 'python-consumer' was very surprised when I learned that
sys.platform is "set" when Python is built - and that platform.system()
was the call to make to get the run-time value. And, of course, they
answered the "question" - what platform am I 'running' on, quite
differently.

* There are several - to many - issues in the past where, in one form or
another, sys.platform, platform.system(), os.name, and I expect more but
cannot think of them atm - used throughout all of Python.

* A Python expert (e.g., core-devs) may understand - inherently - when
each of these is the best option.
As an 'experienced-novice' I am surprised that a "build-time constant"
seems to be preference. This preference is not new (maybe
platform.system() is "new", and os.name() too broad).

* Zooming back to 2012 when "linux3" first appeared this became apparent
that the build-time constant of linux2 was making 'things' difficult
when the packaging was built on linux3 but running with applications
developed on linux2 systems - and early on the move was made to just
have "linux" returned (issue13236). For years the recommendation was
(and maybe is?) to use sys.platform.startswith('linux').

* In between there have been other discussions - and the concern remains
unresolved.

* Personally, not "satisfied", and reluctant to "give up", or "ignore"
something that continues to come up - I opened issue36624 to serve as a
discussion point and worked through what would be needed to have "most"
platforms testing against a CONSTANT defined in test.support - to
provide an example of what it could look like.

* As a non-expert - I EXPECT guidance from the people who know best why
and when the different approaches are preferred. (I am hoping for
something better than 'historical' or 'personal preferences').

* So, using this mail to python-dev to re-open the discussion - my
suggested focus is NOT look at specific (test) code as it is now, but
FOCUS from a perspective of Python3 and beyond - answering the 'concern'
how should 'platform dependencies' be organized'. In other words, do not
focus on an exception (although "listing" perceived exceptions will be
helpful). Focus on "policy" aka "good practice". With that as a starting
point discussing, understanding and validating exceptions to the
guidelines will be much easier.

-- as a 'mentoring' project - I am willing to do as much as I am able to
"implement" something.
My first PR was a 'stab in the dark' and I have already learned "other
things" such as how to git rebase (read merge!). The discussion in the
issue-tracker and PR conversation indicate to me that documentation of
different
approaches to organizing code dependencies is vital. And that it is not
easily available. Devguide is one area - but 'core' documentation is
more important imho - I read, and compare, the Python documentation much
much more often than the devguide docs.

Sincerely,
Michael





signature.asc
Description: OpenPGP digital signature
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YCZYPRBRNOCXLBJ42HA6QZHIBCKZDTGD/

[Python-Dev] Re: radix tree arena map for obmalloc

[Tim Peters]

[Inada Naoki]
>> Increasing pool size is one obvious way to fix these problems.
>> I think 16KiB pool size and 2MiB (huge page size of x86) arena size is
>> a sweet spot for recent web servers (typically, about 32 threads, and
>> 64GiB), but there is no evidence about it.

[Antoine]
> Note that the OS won't give a huge page automatically, because memory
> management becomes much more inflexible then.
>
> For example, the Linux madvise() man page has this to say about
> MADV_HUGEPAGE:
>
>   This feature is primarily aimed at  applications  that
>   use large mappings of data and access large regions of
>   that memory at a time  (e.g.,  virtualization  systems
>   such as QEMU).  It can very easily waste memory (e.g.,
>   a 2 MB mapping that only ever  accesses  1  byte  will
>   result  in  2 MB  of  wired memory instead of one 4 KB
>   page).  See the Linux kernel  source  file  Documenta‐
>   tion/vm/transhuge.txt for more details.
>
> I'm not sure a small objects allocator falls into the right use case
> for huge pages.

The SuperMalloc paper I recently pointed at notes that it uses huge
pages only for "huge" requests.  Not for "small", "medium", or "large"
requests.

But it carves up 2 MiB chunks. aligned at 2 MiB addresses, for each
size class anyway (which use 4K pages).

There are a mix of reasons for that.  Partly for the same reasons I
want bigger pools and arenas:  to stay in the fastest code paths.
Hitting page/arena/chunk boundaries costs cycles for computation and
conditional branches, and clobbers cache lines to access & mutate
bookkeeping info that the fast paths don't touch.

Also to reduce the fraction of allocator space "wasted" on bookkeeping
info.  48 header bytes out of a 4K pool is a bigger percentage hit
than, say, two 4K pages (to hold fancier allocator bookkeeping data
structures) out of a 2M chunk.

And partly for the same reason Neil is keen for bigger arenas in his
branch:  to reduce the size of data structures to keep track of other
bookkeeping info (in Neil's case, a radix tree, which can effectively
shift away the lowest ARENA_BITS bits of addresses it needs to store).

Which hints at much of why it wants "huge" chunks, but doesn't explain
why it doesn't want huge pages except to satisfy huge requests.
That's because it strives to be able to release physical RAM back to
the system on a page basis (which is also part of why it needs fancier
bookkeeping data structures to manage its chunks - it needs to keep
track of which pages are in use, and apply page-based heuristics to
push toward freeing pages).

So that combines very much larger "pools" (2M v 4K) with better
chances of actually returning no-longer-used pages to the system (on a
4K basis rather than a 256K basis).  But it's built on piles of
platform-specific code, and isn't suitable at all for 32-bit boxes
(it' relies on that virtual address space is an abundant resource on
64-bit boxes - reserving 2M of address space is close to trivial, and
could potentially be done millions of times without getting in
trouble).
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CST4C3PF7EGNPEXN3F3LRPGJ7NNDAHTE/

[Python-Dev] Re: How to (best) organize platform dependent blocks of Python code

[Steve Dower]

On 16Jun2019 2354, Michael wrote:

CONCERN: How to organize platform dependent blocks of code
POLICY/RECOMMENDATION: (today) seems to be: arbitrary, although
sys.platform, seems to be the favorite.

* I, as 'python-consumer' was very surprised when I learned that
sys.platform is "set" when Python is built - and that platform.system()
was the call to make to get the run-time value. And, of course, they
answered the "question" - what platform am I 'running' on, quite
differently.

* There are several - to many - issues in the past where, in one form or
another, sys.platform, platform.system(), os.name, and I expect more but
cannot think of them atm - used throughout all of Python.

* A Python expert (e.g., core-devs) may understand - inherently - when
each of these is the best option.
As an 'experienced-novice' I am surprised that a "build-time constant"
seems to be preference. This preference is not new (maybe
platform.system() is "new", and os.name() too broad).


To help focus the discussion a bit, I recommend looking at what we've 
already covered in https://bugs.python.org/issue36624


There really is an apparent lack of consistency in the test suite where 
we use sys.platform, os.name and platform module checks to determine 
whether a test should be run or not. This no doubt extends throughout 
the standard library.


As Michael points out, sys.platform is a specific compile-time constant, 
os.name is a vague compile-time constant, and the platform module 
returns runtime values.


It's entirely possible that the mixed uses are correct, and that we are 
correctly using sys.platform to test things that are based on 
compile-time constants, correctly using os.name to choose between the 
major flavors of posixmodule.c, and so on. But it's not obvious that we 
are doing that, and it's not obvious what the correct choice should be 
in any situation.


We could do with improving the guidance we provide in this area. And as 
Michael says, we are the ones with the knowledge and ability to provide 
that guidance. So this is really a call for opinions on "how should we 
do platform-specific conditions".


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TYARZX6XOFPAHEMUW3G5RUZACRLA64Z5/

[Python-Dev] Re: How to (best) organize platform dependent blocks of Python code

[Guido van Rossum]

I have one data point, perhaps relevant.

In the static type checking world we only care about coarse distinctions.
PEP 484 states that type checkers should understand sys.platform. It does
not require anything else. In practice, this is used almost exclusively to
check for 'win32', since lots of APIs are different on Windows. (In all of
typeshed there's a single check for 'darwin', meaning macOS, vs. currently
89 checks for 'win32'.)

On Mon, Jun 17, 2019 at 9:31 AM Steve Dower  wrote:

> On 16Jun2019 2354, Michael wrote:
> > CONCERN: How to organize platform dependent blocks of code
> > POLICY/RECOMMENDATION: (today) seems to be: arbitrary, although
> > sys.platform, seems to be the favorite.
> >
> > * I, as 'python-consumer' was very surprised when I learned that
> > sys.platform is "set" when Python is built - and that platform.system()
> > was the call to make to get the run-time value. And, of course, they
> > answered the "question" - what platform am I 'running' on, quite
> > differently.
> >
> > * There are several - to many - issues in the past where, in one form or
> > another, sys.platform, platform.system(), os.name, and I expect more but
> > cannot think of them atm - used throughout all of Python.
> >
> > * A Python expert (e.g., core-devs) may understand - inherently - when
> > each of these is the best option.
> > As an 'experienced-novice' I am surprised that a "build-time constant"
> > seems to be preference. This preference is not new (maybe
> > platform.system() is "new", and os.name() too broad).
>
> To help focus the discussion a bit, I recommend looking at what we've
> already covered in https://bugs.python.org/issue36624
>
> There really is an apparent lack of consistency in the test suite where
> we use sys.platform, os.name and platform module checks to determine
> whether a test should be run or not. This no doubt extends throughout
> the standard library.
>
> As Michael points out, sys.platform is a specific compile-time constant,
> os.name is a vague compile-time constant, and the platform module
> returns runtime values.
>
> It's entirely possible that the mixed uses are correct, and that we are
> correctly using sys.platform to test things that are based on
> compile-time constants, correctly using os.name to choose between the
> major flavors of posixmodule.c, and so on. But it's not obvious that we
> are doing that, and it's not obvious what the correct choice should be
> in any situation.
>
> We could do with improving the guidance we provide in this area. And as
> Michael says, we are the ones with the knowledge and ability to provide
> that guidance. So this is really a call for opinions on "how should we
> do platform-specific conditions".
>
> Cheers,
> Steve
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/TYARZX6XOFPAHEMUW3G5RUZACRLA64Z5/
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him/his **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QMFAX2IFHRZX7FMV3Q4YN7N7T3R6E3R6/

[Python-Dev] Re: radix tree arena map for obmalloc

[Tim Peters]

[Tim]
> ...
> Here are some stats from running [memcrunch.py] under
> my PR, but using 200 times the initial number of objects
> as the original script:
>
> n = 2000 #number of things
>
> At the end, with 1M arena and 16K pool:
>
> 3362 arenas * 1048576 bytes/arena  =3,525,312,512
> # bytes in allocated blocks=1,968,233,888
> ...
>  With the larger arenas, none [arenas] were ever released.

...

> BTW, anyone keen to complicate the mmap management should first take
> this recent change into account::
>
> https://bugs.python.org/issue37257
>
> That appears to have killed off _the_ most overwhelmingly common cause
> of obmalloc counter-productively releasing an arena only to create a
> new one again milliseconds later.
>
> My branch, and Neil's, both contain that change, which makes it much
> harder to compare our branches' obmalloc arena stats with 3.7.  It
> turns out that a whole lot of "released arenas" under 3.7 (and will
> still be so in 3.8) were due to that worse-than-useless arena
> thrashing.

To illustrate, I reverted that change in my PR and ran exactly same
thing.  Wow - _then_ the 1M-arena-16K-pool PR reclaimed 1135(!) arenas
instead of none.  Almost all worse than uselessly.  The only one that
"paid" was the last:  the run ended with 3361 arenas still in use
instead of 3362.  Because with the change, one entirely empty arena
remained on the usable_arenas list.

So, word to the wise:  when looking at _debugmallocstats() output, like:

# arenas allocated total   =4,496
# arenas reclaimed =1,135
# arenas highwater mark=3,362
# arenas allocated current =3,361
3361 arenas * 1048576 bytes/arena  =3,524,263,936

the number "reclaimed" isn't really telling us much:  before 3.9, it
may be telling us only how many times obmalloc wasted time on useless
arena thrashing.

The _important_ bit is the difference between "highwater mark" and
"allocated current".  That's how much peak arena address reservation
declined.  In this run, it only managed to release one empty arena
from the peak (which the actual PR does not release, because bpo-37257
changed this to keep (at most) one empty arena available for reuse).
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZFAIAW4VYZRSRG6QJRXTVVTY43562ONI/

[Python-Dev] Re: radix tree arena map for obmalloc

[Tim Peters]

Heh.  I wasn't intending to be nasty, but this program makes our arena
recycling look _much_ worse than memcrunch.py does.  It cycles through
phases.  In each phase, it first creates a large randomish number of
objects, then deletes half of all objects in existence.  Except that
every 10th phase, it deletes 90% instead.  It's written to go through
100 phases, but I killed it after 10 because it was obviously going to
keep on growing without bound.

Note 1:  to do anything deterministic with obmalloc stats these days
appears to require setting the envar PYTHONHASHSEED to 0 before
running (else stats vary even by the time you get to an interactive
prompt).

Note 2:  there are 3 heavily used size classes here, for ints,
2-tuples, and class instances, of byte sizes 32, 64, and 96 on 64-bit
boxes, under my PR and under released 3.7.3.

First with my branch, after phase 10 finishes building objects:

phase 10 adding 9953410
phase 10 has 16743920 objects

# arenas allocated total   =3,114
# arenas reclaimed =0
# arenas highwater mark=3,114
# arenas allocated current =3,114
3114 arenas * 1048576 bytes/arena  =3,265,265,664

# bytes in allocated blocks=3,216,332,784

No arenas have ever been reclaimed, but space utilization is excellent
(about 98.5% of arenas are being used by objects).

Then after phase 10 deletes 90% of everything still alive:

phase 10 deleting factor 90% 15069528
phase 10 done deleting

# arenas allocated total   =3,114
# arenas reclaimed =0
# arenas highwater mark=3,114
# arenas allocated current =3,114
3114 arenas * 1048576 bytes/arena  =3,265,265,664

# bytes in allocated blocks=  323,111,488

Still no arenas have been released, and space utilization is horrid.
A bit less than 10% of allocated space is being use for objects.

Now under 3.7.3.  First when phase 10 is done building:

phase 10 adding 9953410
phase 10 has 16743920 objects

# arenas allocated total   =   14,485
# arenas reclaimed =2,020
# arenas highwater mark=   12,465
# arenas allocated current =   12,465
12465 arenas * 262144 bytes/arena  =3,267,624,960

# bytes in allocated blocks=3,216,219,656

Space utilization is again excellent.  A significant number of arenas
were reclaimed - but usefully?  Let's see how things turn out after
phase 10 ends deleting 90% of the objects:

phase 10 deleting factor 90% 15069528
phase 10 done deleting

# arenas allocated total   =   14,485
# arenas reclaimed =2,020
# arenas highwater mark=   12,465
# arenas allocated current =   12,465
12465 arenas * 262144 bytes/arena  =3,267,624,960

# bytes in allocated blocks=  322,998,360

Didn't manage to reclaim anything!  Space utililization is again
horrid, and it's actually consuming a bit more arena bytes than when
running under the PR.

Which is just more of what I've been seeing over & over:  3.7.3 and
the PR both do a fine job of recycling arenas, or a horrid job,
depending on the program.

For excellent recycling, change this program to use a dict instead of a set.  So

data = {}

at the start, fill it with

data[serial] = Stuff()

and change

data.pop()

to use .popitem().

The difference is that set elements still appear in pseudo-random
order, but dicts are in insertion-time order.  So data.popitem() loses
the most recently added dict entry, and the program is then just
modeling stack allocation/deallocation.

def doit():
import random
from random import randrange
import sys

class Stuff:
# add cruft so it takes 96 bytes under 3.7 and 3.8
__slots__ = tuple("abcdefg")

def __hash__(self):
return hash(id(self))

LO = 5_000_000
HI = LO * 2
data = set()
serial = 0
random.seed(42)

for phase in range(1, 101):
toadd = randrange(LO, HI)
print("phase", phase, "adding", toadd)
for _ in range(toadd):
data.add((serial, Stuff()))
serial += 1
print("phase", phase, "has", len(data), "objects")
sys._debugmallocstats()
factor = 0.5 if phase % 10 else 0.9
todelete = int(len(data) * factor)
print(f"phase {phase} deleting factor {factor:.0%} {todelete}")
for _ in range(todelete):
data.pop()
print("phase", phase, "done deleting")
sys._debugmallocstats()

doit()
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/list

[Python-Dev] Re: radix tree arena map for obmalloc

[Tim Peters]

[Tim]
> ...
> Now under 3.7.3.  First when phase 10 is done building:
>
> phase 10 adding 9953410
> phase 10 has 16743920 objects
>
> # arenas allocated total   =   14,485
> # arenas reclaimed =2,020
> # arenas highwater mark=   12,465
> # arenas allocated current =   12,465
> 12465 arenas * 262144 bytes/arena  =3,267,624,960
>
> # bytes in allocated blocks=3,216,219,656
>
> Space utilization is again excellent.  A significant number of arenas
> were reclaimed - but usefully?

Nope!  Digging through all the output, all the arena recycling
happened in the "_add_ millions of objects" stages, never in the
"delete millions of objects" stages.  So it was just more accidental
arena thrashing (which was recently repaired in bpo-37257).
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JXQHYTTFKBE27DKZ532ENIMPNPE7PJ44/