[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

STINNER Victor added the comment: It is possible to install a custom hook on memory allocators using the PEP 445: see for example attached fatalmalloc.c. I don't see much traction to get this feature since 2013, I reject the feature request. -- resolution: -> rejected stage: patch

[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

Changes by STINNER Victor : Added file: http://bugs.python.org/file32042/setup.py ___ Python tracker ___ ___ Python-bugs-list mailing list Uns

[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

STINNER Victor added the comment: This issue is maybe a new usecase of the PEP 445. Try attached Python module fatalmalloc.c, use attached setup.py script to build it. $ python3.4 -c 'import fatalmalloc; x="x"*(50*1024*1024); print(len(x))' Traceback (most recent call last): File "", line 1,

[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

STINNER Victor added the comment: See also issue #1195571: simple callback system for Py_FatalError. It is somehow related. -- ___ Python tracker ___ ___

[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

Antoine Pitrou added the comment: If we want to get more sophisticated, I would suggest a two-pronged approach: - first call a user-defined oom function - if calling the user-defined oom function raises MemoryError, dump a fatal error -- nosy: +pitrou __

[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

Changes by STINNER Victor : -- nosy: +haypo ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python

[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

Lukas Lueg added the comment: Another proposal: Add a new BaseClass that, if inherited from, causes an exception to be uncatchable (e.g. class HardMemoryError(MemoryError, UncatchableException)). -- ___ Python tracker

[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

Changes by Christian Theune : -- nosy: +barry ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.pytho

[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

Christian Theune added the comment: I feel unsure how to help this move along. I agree that making it possible for applications to carefully work with MemoryErrors is a good idea. I don't think heuristics to determine which situation we are in will solve this but make it more spooky. (This is

[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

Changes by Jesús Cea Avión : -- nosy: +jcea ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.

[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

Lukas Lueg added the comment: The heuristic basically has to decide if memory pressure is so high that it's not save to return to the interpreter. Even if there is a chosen value (e.g. failed allocation attempts below 1mb are considered fatal), there can always be another OS-thread in the inte

[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

Christian Heimes added the comment: We can create some kind of heuristic when the memory error handler knows the size of the new block and the size difference on realloc(). It could be an application specific threshold, too. -- ___ Python tracker <

[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

Christian Heimes added the comment: Of course it can be fixed. But it's going to be a long and tedious piece of work. PyErr_NoMemory() is called 314 times. MemoryError is raised about 40 times in C code. I'm not sure what your question about realloc() really is. Are you unsure how a a failing

[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

Lukas Lueg added the comment: In any strategy only a heuristic could be used in order to decide wether or not it's safe to raise MemoryError. How exactly is memory pressure expected for x=[2]*200 but not for x=2*200 ? I don't think a new function could ultimatly achieve it's goal. If MemoryErr

[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

Guido van Rossum added the comment: Well, you can fix that, right? Just add a new function with a better signature and use that for one of the two scenarios. (I think third scenario might be when realloc fails -- IIRC it doesn't guarantee that the original pointer is still valid either?) -

[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

Christian Heimes added the comment: MemoryError can be raised under two different circumstances that should be handled differently. Either the program tries to allocate a rather large chunk of memory for e.g. a string with a couple of hundred KB and more. Or Python can't malloc() even small am

[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

Lukas Lueg added the comment: I have to say this feels like spooky action at a distance. Wouldnt it be less intrusive - while achieving the same result - to make MemoryError uncatchable if the flag is set? -- nosy: +ebfe ___ Python tracker

[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

Guido van Rossum added the comment: There used to be some places in the C code that raise MemoryError to indicate that some parameter indicating a desired result size is out of range, i.e. before even trying to allocate anything. I can't confirm any, but these should probably be replaced with ano

[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

Benjamin Peterson added the comment: Where it is now means raising a MemoryError in Python code will cause the fatal error. I don't know if that's a good idea. -- ___ Python tracker ___

[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

Christian Theune added the comment: I pondered PyErr_NoMemory as well. However, I noticed not all locations in Python use PyErr_NoMemory to raise a MemoryError, and I'm also afraid that external libraries will have the same problem. Can you explain why you consider PyErr_NoMemory to be the bet

[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

Changes by Hynek Schlawack : -- nosy: +hynek ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python

[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

Benjamin Peterson added the comment: I think the fatal erroring should be done in PyErr_NoMemory. -- nosy: +benjamin.peterson ___ Python tracker ___ _

[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

Christian Heimes added the comment: Thanks! Py_FatalError() might be too drastic for the task. It calls abort() which kills the process with SIGABRT. The function closes and flushes all stream but no additional cleanup code is executed. This might be bad for resources like shared memories, na

[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

Changes by Christian Theune : Removed file: http://bugs.python.org/file27821/9430a5c65114.diff ___ Python tracker ___ ___ Python-bugs-list mai

[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

Christian Theune added the comment: Grr. Sorry. The automatic patch extraction went wrong and I didn't notice. Here's a manual try. -- Added file: http://bugs.python.org/file27822/issue16381.diff ___ Python tracker

[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

Christian Heimes added the comment: Your proposal sounds like a very good idea. IMHO it should be discussed on the python-ideas or python-dev mailing list before it gets integrated into 3.4. Embrace yourself for some serious bike shedding! :) By the way your patch contains several changes that

[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

Changes by Christian Theune : -- keywords: +patch Added file: http://bugs.python.org/file27821/9430a5c65114.diff ___ Python tracker ___ __

[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

New submission from Christian Theune: I run long-running server processes (web apps, etc) a lot and I keep encountering the situation that many applications will not properly deal with MemoryError exceptions but end up in an unusable state. >From an operational perspective I wish the process i