[Python-Dev] Re: radix tree arena map for obmalloc

2019-06-17 Thread Tim Peters
[Tim] > ... > Now under 3.7.3. First when phase 10 is done building: > > phase 10 adding 9953410 > phase 10 has 16743920 objects > > # arenas allocated total = 14,485 > # arenas reclaimed =2,020 > # arenas highwater mark=

[Python-Dev] Re: radix tree arena map for obmalloc

2019-06-17 Thread Tim Peters
Heh. I wasn't intending to be nasty, but this program makes our arena recycling look _much_ worse than memcrunch.py does. It cycles through phases. In each phase, it first creates a large randomish number of objects, then deletes half of all objects in existence. Except that every 10th phase,

[Python-Dev] Re: radix tree arena map for obmalloc

2019-06-17 Thread Tim Peters
[Tim] > ... > Here are some stats from running [memcrunch.py] under > my PR, but using 200 times the initial number of objects > as the original script: > > n = 2000 #number of things > > At the end, with 1M arena and 16K pool: > > 3362 arenas * 1048576 bytes/arena =3,525,312,512 > #

[Python-Dev] Re: How to (best) organize platform dependent blocks of Python code

2019-06-17 Thread Guido van Rossum
I have one data point, perhaps relevant. In the static type checking world we only care about coarse distinctions. PEP 484 states that type checkers should understand sys.platform. It does not require anything else. In practice, this is used almost exclusively to check for 'win32', since lots of

[Python-Dev] Re: How to (best) organize platform dependent blocks of Python code

2019-06-17 Thread Steve Dower
On 16Jun2019 2354, Michael wrote: CONCERN: How to organize platform dependent blocks of code POLICY/RECOMMENDATION: (today) seems to be: arbitrary, although sys.platform, seems to be the favorite. * I, as 'python-consumer' was very surprised when I learned that sys.platform is "set" when Python

[Python-Dev] Re: radix tree arena map for obmalloc

2019-06-17 Thread Tim Peters
[Inada Naoki] >> Increasing pool size is one obvious way to fix these problems. >> I think 16KiB pool size and 2MiB (huge page size of x86) arena size is >> a sweet spot for recent web servers (typically, about 32 threads, and >> 64GiB), but there is no evidence about it. [Antoine] > Note that

[Python-Dev] How to (best) organize platform dependent blocks of Python code

2019-06-17 Thread Michael
CONCERN: How to organize platform dependent blocks of code POLICY/RECOMMENDATION: (today) seems to be: arbitrary, although sys.platform, seems to be the favorite. * I, as 'python-consumer' was very surprised when I learned that sys.platform is "set" when Python is built - and that

[Python-Dev] Re: radix tree arena map for obmalloc

2019-06-17 Thread Inada Naoki
On Mon, Jun 17, 2019 at 6:14 PM Antoine Pitrou wrote: > But it's not enabled by default... And there are reasons for that (see > the manpage I quoted). Uh, then if people want to use huge page, they need to enable it on system wide, or add madvice in obmalloc.c. > > In web applications, it's

[Python-Dev] Re: radix tree arena map for obmalloc

2019-06-17 Thread Antoine Pitrou
Le 17/06/2019 à 10:55, Inada Naoki a écrit : > On Mon, Jun 17, 2019 at 5:18 PM Antoine Pitrou wrote: >> >> On Mon, 17 Jun 2019 11:15:29 +0900 >> Inada Naoki wrote: >>> >>> Increasing pool size is one obvious way to fix these problems. >>> I think 16KiB pool size and 2MiB (huge page size of x86)

[Python-Dev] Re: radix tree arena map for obmalloc

2019-06-17 Thread Inada Naoki
On Mon, Jun 17, 2019 at 5:18 PM Antoine Pitrou wrote: > > On Mon, 17 Jun 2019 11:15:29 +0900 > Inada Naoki wrote: > > > > Increasing pool size is one obvious way to fix these problems. > > I think 16KiB pool size and 2MiB (huge page size of x86) arena size is > > a sweet spot for recent web

[Python-Dev] Re: radix tree arena map for obmalloc

2019-06-17 Thread Antoine Pitrou
On Mon, 17 Jun 2019 11:15:29 +0900 Inada Naoki wrote: > > Increasing pool size is one obvious way to fix these problems. > I think 16KiB pool size and 2MiB (huge page size of x86) arena size is > a sweet spot for recent web servers (typically, about 32 threads, and > 64GiB), but there is no