[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

[STINNER Victor]

STINNER Victor  added the comment:

It is possible to install a custom hook on memory allocators using the PEP 445: 
see for example attached fatalmalloc.c.

I don't see much traction to get this feature since 2013, I reject the feature 
request.

--
resolution:  -> rejected
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

[STINNER Victor]

STINNER Victor added the comment:

This issue is maybe a new usecase of the PEP 445.

Try attached Python module fatalmalloc.c, use attached setup.py script to build 
it.

$ python3.4 -c 'import fatalmalloc; x=x*(50*1024*1024); print(len(x))'
Traceback (most recent call last):
  File string, line 1, in module
MemoryError

$ python3.4 -c 'import fatalmalloc; fatalmalloc.enable(); x=x*(50*1024*1024)'
$ echo $?
1

--
Added file: http://bugs.python.org/file32041/fatalmalloc.c

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16381
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

[STINNER Victor]

Changes by STINNER Victor victor.stin...@gmail.com:


Added file: http://bugs.python.org/file32042/setup.py

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16381
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

[STINNER Victor]

Changes by STINNER Victor victor.stin...@gmail.com:


--
nosy: +haypo

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16381
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

[Antoine Pitrou]

Antoine Pitrou added the comment:

If we want to get more sophisticated, I would suggest a two-pronged approach:
- first call a user-defined oom function
- if calling the user-defined oom function raises MemoryError, dump a fatal 
error

--
nosy: +pitrou

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16381
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

[STINNER Victor]

STINNER Victor added the comment:

See also issue #1195571: simple callback system for Py_FatalError. It is 
somehow related.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16381
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

[Lukas Lueg]

Lukas Lueg added the comment:

Another proposal: Add a new BaseClass that, if inherited from, causes an 
exception to be uncatchable (e.g. class HardMemoryError(MemoryError, 
UncatchableException)).

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16381
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

[Christian Theune]

Christian Theune added the comment:

I feel unsure how to help this move along. 

I agree that making it possible for applications to carefully work with 
MemoryErrors is a good idea. I don't think heuristics to determine which 
situation we are in will solve this but make it more spooky. (This is a 
semantic issue anyways).

My goal is to use the flag to make it easier for operators to deal with 
applications that don't get it right. This isn't to defend bad programming but 
to support temporary relief.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16381
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

[Christian Theune]

Changes by Christian Theune c...@gocept.com:


--
nosy: +barry

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16381
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

[Jesús Cea Avión]

Changes by Jesús Cea Avión j...@jcea.es:


--
nosy: +jcea

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16381
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

[Hynek Schlawack]

Changes by Hynek Schlawack h...@ox.cx:


--
nosy: +hynek

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16381
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

[Christian Theune]

Christian Theune added the comment:

I pondered PyErr_NoMemory as well. However, I noticed not all locations in 
Python use PyErr_NoMemory to raise a MemoryError, and I'm also afraid that 
external libraries will have the same problem.

Can you explain why you consider PyErr_NoMemory to be the better option? (I can 
think of performance but I wouldn't want to trade it in against reliability.)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16381
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

[Benjamin Peterson]

Benjamin Peterson added the comment:

Where it is now means raising a MemoryError in Python code will cause the fatal 
error. I don't know if that's a good idea.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16381
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

[Guido van Rossum]

Guido van Rossum added the comment:

There used to be some places in the C code that raise MemoryError to
indicate that some parameter indicating a desired result size is out
of range, i.e. before even trying to allocate anything. I can't
confirm any, but these should probably be replaced with another
exception. (I thought string times integer was one such place, but it
raises OverflowError now.) If there are any left those should probably
be replaced with OverflowError.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16381
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

[Lukas Lueg]

Lukas Lueg added the comment:

I have to say this feels like spooky action at a distance.

Wouldnt it be less intrusive - while achieving the same result - to make 
MemoryError uncatchable if the flag is set?

--
nosy: +ebfe

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16381
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

[Christian Heimes]

Christian Heimes added the comment:

MemoryError can be raised under two different circumstances that should be 
handled differently. Either the program tries to allocate a rather large chunk 
of memory for e.g. a string with a couple of hundred KB and more. Or Python 
can't malloc() even small amounts of memory for its most basic operations like 
raising an exception object. That's why PyErr_NoMemory() exists and why it uses 
a pre-allocated MemoryError object.

When a program can't allocate memory for an image it usually has enough memory 
left to do proper error reporting and shut down. However when even x = 2 * 
200 fails with a memory error then proper shutdown will most likely fail and 
hang up the process, too.

Too bad that PyErr_NoMemory() is used for both scenarios and isn't aware how 
much memory was requested.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16381
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

[Guido van Rossum]

Guido van Rossum added the comment:

Well, you can fix that, right? Just add a new function with a better
signature and use that for one of the two scenarios.

(I think  third scenario might be when realloc fails -- IIRC it
doesn't guarantee that the original pointer is still valid either?)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16381
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

[Lukas Lueg]

Lukas Lueg added the comment:

In any strategy only a heuristic could be used in order to decide wether or not 
it's safe to raise MemoryError. How exactly is memory pressure expected for 
x=[2]*200 but not for x=2*200 ?

I don't think a new function could ultimatly achieve it's goal. If MemoryError 
occurs, all bets are off. Shutting down as fast as possible is the best we 
could do, if requested.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16381
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

[Christian Heimes]

Christian Heimes added the comment:

Of course it can be fixed. But it's going to be a long and tedious piece of 
work. PyErr_NoMemory() is called 314 times. MemoryError is raised about 40 
times in C code.

I'm not sure what your question about realloc() really is. Are you unsure how a 
a failing realloc() calls must be resolved?

Code like

  ptr = realloc(ptr, size);

is usually a memory leak as it leaks the previously allocated memory in ptr. In 
the error case the block in ptr is left untouched and is still valid. It must 
be freed manually or can be used further and freed later.

  if((ptr2 == realloc(ptr, size)) == NULL) {
free(ptr);
  }
  else {
ptr = ptr2;
  }

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16381
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

[Christian Heimes]

Christian Heimes added the comment:

We can create some kind of heuristic when the memory error handler knows the 
size of the new block and the size difference on realloc(). It could be an 
application specific threshold, too.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16381
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

[Lukas Lueg]

Lukas Lueg added the comment:

The heuristic basically has to decide if memory pressure is so high that it's 
not save to return to the interpreter. Even if there is a chosen value (e.g. 
failed allocation attempts below 1mb are considered fatal), there can always be 
another OS-thread in the interpreter process that eats away exactly that memory 
while we are returning MemoryError - the program might still hang.
FWICS all MemoryErrors are to be considered fatal or none of them

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16381
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

[Christian Theune]

New submission from Christian Theune:

I run long-running server processes (web apps, etc) a lot and I keep 
encountering the situation that many applications will not properly deal with 
MemoryError exceptions but end up in an unusable state.

From an operational perspective I wish the process in this case would just 
fail and exit.

I talked to Guido about this general idea at EuroPython2012 and he encouraged 
me to look into this. 

Here's a patch:
https://bitbucket.org/ctheune/cpython/changeset/323bb572344d46df669d3dbec4431cf6720fc5b4

I think it does what I want it to do, but a) my C knowledge is really bad and 
b) I'm not sure whether this is the right approach anyway.

I'd appreciate feedback and possibly inclusion in the core.

--
components: Interpreter Core
hgrepos: 158
messages: 174413
nosy: ctheune, gvanrossum
priority: normal
severity: normal
status: open
title: Introduce option to force the interpreter to exit upon MemoryErrors
type: enhancement
versions: Python 3.4

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16381
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

[Christian Theune]

Changes by Christian Theune c...@gocept.com:


--
keywords: +patch
Added file: http://bugs.python.org/file27821/9430a5c65114.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16381
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

[Christian Heimes]

Christian Heimes added the comment:

Your proposal sounds like a very good idea. IMHO it should be discussed on the 
python-ideas or python-dev mailing list before it gets integrated into 3.4. 
Embrace yourself for some serious bike shedding! :)

By the way your patch contains several changes that aren't related to your 
proposal. Can you clean up your patch, please? It makes code review much easier.

--
nosy: +christian.heimes
stage:  - patch review

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16381
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

[Christian Theune]

Christian Theune added the comment:

Grr. Sorry. The automatic patch extraction went wrong and I didn't notice. 
Here's a manual try.

--
Added file: http://bugs.python.org/file27822/issue16381.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16381
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

[Christian Theune]

Changes by Christian Theune c...@gocept.com:


Removed file: http://bugs.python.org/file27821/9430a5c65114.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16381
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

[Christian Heimes]

Christian Heimes added the comment:

Thanks!

Py_FatalError() might be too drastic for the task. It calls abort() which kills 
the process with SIGABRT. The function closes and flushes all stream but no 
additional cleanup code is executed. This might be bad for resources like 
shared memories, named semaphores or database connections. On Windows abort() 
causes a pop up window with a crash report, too.

Would exit(3) or _exit(2) work here?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16381
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16381] Introduce option to force the interpreter to exit upon MemoryErrors

[Benjamin Peterson]

Benjamin Peterson added the comment:

I think the fatal erroring should be done in PyErr_NoMemory.

--
nosy: +benjamin.peterson

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16381
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com