It happened very accidentally. Since venv is used,
many paths in the interpreter is changed. So how memory
is used are changed.
Let's reproduce the accident.
$ cat m2.py
import pickle, sys
LIST = pickle.dumps([[0]*10 for _ in range(10)], pickle.HIGHEST_PROTOCOL)
N = 1000
z = [[0]*10 for _ in range(N)]
if '-c' in sys.argv:
sys._debugmallocstats()
sys.exit()
for _ in range(100000):
pickle.loads(LIST)
$ /usr/bin/time python3 m2.py
0.42user 0.00system 0:00.43elapsed 99%CPU (0avgtext+0avgdata
9100maxresident)k
0inputs+0outputs (0major+1139minor)pagefaults 0swaps
There are only 1139 faults. It is less than 100000.
$ /usr/bin/time python3 m2.py -c
...
14 unused pools * 4096 bytes = 57,344
...
adjust N im m2.py until it shows "0 unused pools".
In my case, N=1390.
$ /usr/bin/time python3 m2.py
0.51user 0.33system 0:00.85elapsed 99%CPU (0avgtext+0avgdata
9140maxresident)k
0inputs+0outputs (0major+201149minor)pagefaults 0swaps
200000 faults!
It seems two page fault / loop. (2 pools are used and returned).
On Wed, Feb 27, 2019 at 7:51 PM Victor Stinner <vstin...@redhat.com> wrote:
> Sorry, I didn't get a coffee yet: more *often* in a venv.
>
> Le mer. 27 févr. 2019 à 11:32, Victor Stinner <vstin...@redhat.com> a
> écrit :
> >
> > Any idea why Python calls mmap+munmap more even in a venv?
> >
> > Victor
> >
> > Le mer. 27 févr. 2019 à 10:00, INADA Naoki <songofaca...@gmail.com> a
> écrit :
> > >
> > > >
> > > > > Ah, another interesting point, this huge slowdown happens only
> when bm_pickle.py
> > > > > is executed through pyperformance. When run it directly, slowdown
> is
> > > > > not so large.
> > > >
> > > > pyperformance runs benchmarks in a virtual environment. I don't know
> > > > if it has any impact on bm_pickle.
> > > >
> > > > Most pyperformance can be run outside a virtual env if required
> > > > modules are installed on the system. (bm_pickle only requires the
> > > > stdlib and perf.)
> > > >
> > >
> > > Bingo!
> > >
> > > Without venv:
> > >
> > > unpickle: Mean +- std dev: 26.9 us +- 0.0 us
> > > % time seconds usecs/call calls errors syscall
> > > ------ ----------- ----------- --------- --------- ----------------
> > > 28.78 0.000438 0 1440 read
> > > 27.33 0.000416 1 440 25 stat
> > > 9.72 0.000148 1 144 mmap
> > > ...
> > > 0.79 0.000012 1 11 munmap
> > >
> > > With venv:
> > >
> > > % time seconds usecs/call calls errors syscall
> > > ------ ----------- ----------- --------- --------- ----------------
> > > 57.12 0.099023 2 61471 munmap
> > > 41.87 0.072580 1 61618 mmap
> > > 0.23 0.000395 1 465 27 stat
> > >
> > > unpickle and unpickle_list creates massive same-sized objects, then
> all objects are
> > > removed. If all pools in the arena is freed, munmap is called.
> > >
> > > I think we should save some arenas to reuse. On recent Linux,
> > > we may be able to use MADV_FREE instead of munmap.
> > >
> >
> >
> > --
> > Night gathers, and now my watch begins. It shall not end until my death.
>
>
>
> --
> Night gathers, and now my watch begins. It shall not end until my death.
>
--
INADA Naoki <songofaca...@gmail.com>
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
