Serhiy Storchaka added the comment:
Instead of the loop you can use writelines():
f.writelines([b'\0' * bs] * (size // bs))
It would be nice to add a comment that estimate why os.ftruncate() or
seek+write can't be used here. At least a link to this issue with short
estimation.
Antoine Pitrou added the comment:
Actually, recent POSIX states unconditionally that:
« If the file previously was smaller than this size, ftruncate() shall increase
the size of the file. If the file size is increased, the extended area shall
appear as if it were zero-filled. »
(from
Changes by Antoine Pitrou pit...@free.fr:
--
versions: -Python 3.4
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21116
___
___
Python-bugs-list
Antoine Pitrou added the comment:
Ok, I've committed the patch. If desired, the generic API for shared memory can
be tackled in a separate issue. Thank you Médéric!
--
resolution: - fixed
stage: patch review - resolved
status: open - closed
___
Roundup Robot added the comment:
New changeset 0f944e424d67 by Antoine Pitrou in branch 'default':
Issue #21116: Avoid blowing memory when allocating a multiprocessing shared
https://hg.python.org/cpython/rev/0f944e424d67
--
nosy: +python-dev
___
Charles-François Natali added the comment:
Indeed, I think it would make sense to consider this for 3.4, and even 2.7
if we opt for a simple fix.
As for the best way to fix it in the meantime, I'm fine with a buffered
zero-filling (the mere fact that noone ever complained until now probably
Médéric Boquien added the comment:
Thanks for the explanations Charles-François. I guess the new API would not be
before 3.5 at least. Is there still a chance to integrate my patch (or any
other) to improve the situation for the 3.4 series though?
--
Charles-François Natali added the comment:
If I remember correctly the problem is that some OS like linux (and
probably others) do not really allocate space until something is written.
If that's the case then the process may get killed later on when it writes
something in the array.
Yes, it's
Médéric Boquien added the comment:
the process will get killed when first writing to the page in case of memory
pressure.
According to the documentation, the returned shared array is zeroed.
https://docs.python.org/3.4/library/multiprocessing.html#module-multiprocessing.sharedctypes
In that
Charles-François Natali added the comment:
Also, the FreeBSD man page for mmap() has the following warning:
That's mostly important for real file-backed mapping.
In our case, we don't want a file-backed mmap: we expect the mapping to fit
entirely in memory, so the writeback/read performance
Antoine Pitrou added the comment:
Or maybe it's time to add an API to access shared memory from Python
(since
that's really what we're trying to achieve here).
That sounds like a good idea. Especially since we now have the memoryview type.
--
___
Charles-François Natali added the comment:
Zero-filling mmap's backing file isn't really optimal: why not use truncate()
instead? This way, it'll avoid completely I/O on filesystems that support
sparse files, and should still work on FS that don't.
--
nosy: +neologix
Médéric Boquien added the comment:
If I remember correctly the problem is that some OS like linux (and probably
others) do not really allocate space until something is written. If that's the
case then the process may get killed later on when it writes something in the
array.
Here is a quick
Richard Oudkerk added the comment:
Using truncate() to zero extend is not really portable: it is only guaranteed
on XSI-compliant POSIX systems.
Also, the FreeBSD man page for mmap() has the following warning:
WARNING! Extending a file with ftruncate(2), thus creating a big
hole, and then
Médéric Boquien added the comment:
I have now signed the contributor's agreement.
As for the unit test I was looking at it. However, I was wondering how to write
a test that would have triggered the problem. It only shows up for very large
arrays and it depends on occupied memory and the
New submission from Médéric Boquien:
It is currently impossible to create multiprocessing shared arrays larger than
50% of memory size under linux (and I assume other unices). A simple test case
would be the following:
from multiprocessing.sharedctypes import RawArray
import ctypes
foo =
Médéric Boquien added the comment:
Updated the patch not to create a uselessly large array if the size is small
than the block size.
--
Added file: http://bugs.python.org/file34686/shared_array.diff
___
Python tracker rep...@bugs.python.org
Changes by Médéric Boquien mboqu...@free.fr:
Removed file: http://bugs.python.org/file34685/shared_array.diff
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21116
___
Changes by Antoine Pitrou pit...@free.fr:
--
nosy: +sbt
stage: - patch review
type: - resource usage
versions: +Python 3.5
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21116
___
Médéric Boquien added the comment:
New update of the patch following Antoine Pitrou's comments. PEP8 does not
complain anymore.
--
Added file: http://bugs.python.org/file34687/shared_array.diff
___
Python tracker rep...@bugs.python.org
Antoine Pitrou added the comment:
You overlooked the part where I was suggesting to add a unit test :-)
Also, you'll have to sign a contributor's agreement at
https://www.python.org/psf/contrib/contrib-form/
Thanks!
--
nosy: +pitrou
___
Python
21 matches
Mail list logo
