[issue27535] Memory leaks when opening tons of files

[Jonathan Bastien-Filiatrault]

Jonathan Bastien-Filiatrault  added the comment:

@vstinner Yes, from what I saw, the leak was from the registry / deduplication 
logic.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27535] Memory leaks when opening tons of files

[STINNER Victor]

STINNER Victor  added the comment:

I created PR 4489 to fix the memory leak for the "ignore "action". IMHO it's 
interesting to modify the "ignore" action because Python uses ignore many 
warnings by default:

haypo@selma$ python3 -c 'import pprint, warnings; 
pprint.pprint(warnings.filters)'
[('ignore', None, , None, 0),
 ('ignore', None, , None, 0),
 ('ignore', None, , None, 0),
 ('ignore', None, , None, 0),
 ('ignore', None, , None, 0)]


I created memory2.py (based on attached memory.py) to measure the leak.

Currently, the "always" action doesn't leak, but "ignore" and "default" actions 
leak:

haypo@selma$ ./python -W always memory2.py 
Memory peak grow: +3.2 kB
Warning filters:
[('always',
  re.compile('', re.IGNORECASE),
  ,
  re.compile(''),
  0),
 ('ignore', None, , None, 0),
 ('always', None, , None, 0)]

haypo@selma$ ./python -W ignore memory2.py 
Memory peak grow: +26692.3 kB
Warning filters:
[('ignore',
  re.compile('', re.IGNORECASE),
  ,
  re.compile(''),
  0),
 ('ignore', None, , None, 0),
 ('always', None, , None, 0)]

haypo@selma$ ./python -W default memory2.py 
Memory peak grow: +26693.2 kB
Warning filters:
[('default',
  re.compile('', re.IGNORECASE),
  ,
  re.compile(''),
  0),
 ('ignore', None, , None, 0),
 ('always', None, , None, 0)]


With my PR 4489, the "ignore" action doesn't leak anymore:

haypo@selma$ ./python -W ignore memory2.py 
Memory peak grow: +2.4 kB
Warning filters:
[('ignore',
  re.compile('', re.IGNORECASE),
  ,
  re.compile(''),
  0),
 ('ignore', None, , None, 0),
 ('always', None, , None, 0)]

--
Added file: https://bugs.python.org/file47281/memory2.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27535] Memory leaks when opening tons of files

[STINNER Victor]

Change by STINNER Victor :


--
keywords: +patch
pull_requests: +4427
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27535] Memory leaks when opening tons of files

[STINNER Victor]

STINNER Victor  added the comment:

The problem is not the function displaying the warning, but warnings registries 
(__warningregistry__) no?

The behaviour depends on the action of the filter matching the ResourceWarning 
warning:

* always: don't touch the registry => no leak
* ignore: add the key to registry => leak!
* another action: add the key to registry => leak!

The key is the tuple (text, category, lineno).

I understand why always doesn't touch the registry. But why does "ignore" 
action touch the registry? Not only the user will not see the message, but the 
memory will slowly grow in the background.

--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27535] Memory leaks when opening tons of files

[Jonathan Bastien-Filiatrault]

Jonathan Bastien-Filiatrault  added the comment:

We just got hit by this. We had one specific case where files with unique names 
were not being closed and Python was leaking a lot of memory over a few days.

--
nosy: +Jonathan Bastien-Filiatrault

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27535] Memory leaks when opening tons of files

[Raymond Hettinger]

Raymond Hettinger added the comment:

I would like to see this fixed.  It is rather unfortunate that a tool designed 
to help find possible resource leaks is itself a certain resource leak.

--
nosy: +rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27535] Memory leaks when opening tons of files

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Thus, the problem is that even ignored warnings are saved. If there is a simple 
fix of this problem we can fix it. But if there is no simple and 
straightforward way, it may be not worth to complicate the code for such case. 
I'll look at the code.

--
assignee:  -> serhiy.storchaka
priority: normal -> low
versions: +Python 3.6 -Python 3.4, Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27535] Memory leaks when opening tons of files

[R. David Murray]

R. David Murray added the comment:

I recommend rejecting this.  Properly closing flies is the correct programming 
habit, which is why the ResourceWarning exists.  The Pillow example seems to 
just indicate that people using Pillow need to pay attention to the 
ResourceWarning and close the files they open with pillow.

--
nosy: +r.david.murray

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27535] Memory leaks when opening tons of files

[Александр Карпинский]

Александр Карпинский added the comment:

@serhiy.storchaka

Any filters not solves the problem because warnings module SAVES EVERY WARNING 
MESSAGE for further duplication checks.

Yes, the file name is helpful for determining the source of the POSSIBLE leak. 
While file name in error message IS SOURCE OF THE LEAK. When you aware about 
not closed files, you can log file names. You have a way to do that. But if you 
want to ignore the warning, the memory will leak. Not because of not closed 
files, but because of warning itself.

I do want to close files in my script, but in Pillow, it is not so easy due to 
backward compatibility.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27535] Memory leaks when opening tons of files

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

The file name is helpful for determining the source of the leak. If you don't 
want to close files in your script, set an appropriate warning filter to 
silence this kind of warnings.

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27535] Memory leaks when opening tons of files

[Александр Карпинский]

New submission from Александр Карпинский:

Actually, this issue is related to the warning module. The test script creates 
a lot of files with different names and deletes them. On the first pass the 
scripts calls the `f.close()` method for every file. On the second it doesn't.

As a result, on the first pass, the memory consumption is constant and is about 
3.9 Mb for my environment. For the second pass, the memory consumption 
constantly grows up to 246 Mb for 1 million files. I.e. memory leak is about 
254 bytes for every opened file.

This happens because of warning about not closed files. The memory is consumed 
for storing all possible warning messages generated for different file names to 
prevent further warnings with the same messages. Which is not the case if we 
are always opening new files.

While not calling f.close() **might** lead to memory issues, the warning which 
notifies about this **absolutely guaranteed** leads to memory issue. Although 
f.close() is highly recommended, this is still the issue for code which doesn't 
use it for some reasons and doesn't experience other problems with not closed 
files because files are actually always closed in IOBase.__del__.

This issue was discovered in this report: 
https://github.com/python-pillow/Pillow/issues/2019


The best way to fix this is excluding file name from warning message. As a 
result, the message for every file will be the same and warnings registry will 
not grow. For now, the message looks like this:

./memory.py:23: ResourceWarning: unclosed file <_io.FileIO name='__999.tmp' 
mode='wb' closefd=True>
  open_files(number, close)

It may looks like this:

./memory.py:23: ResourceWarning: unclosed file 
  open_files(number, close)

--
components: IO
files: memory.py
messages: 270600
nosy: Александр Карпинский
priority: normal
severity: normal
status: open
title: Memory leaks when opening tons of files
type: resource usage
versions: Python 3.4, Python 3.5
Added file: http://bugs.python.org/file43762/memory.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com