[issue29638] Spurious failures in test_collections in releak hunting mode after typing is imported

[Serhiy Storchaka]

Serhiy Storchaka added the comment:


New changeset b414e349eb6a6140a81ccec249bae3abe939e836 by Serhiy Storchaka 
(Ivan Levkivskyi) in branch '3.6':
bpo-29638: Fix spurious refleaks after typing is imported (#469) (#483)
https://github.com/python/cpython/commit/b414e349eb6a6140a81ccec249bae3abe939e836


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29638] Spurious failures in test_collections in releak hunting mode after typing is imported

[Serhiy Storchaka]

Serhiy Storchaka added the comment:


New changeset 7acffa23c9e7866e5b4b98b8f3c28b14ec1c688c by Serhiy Storchaka 
(Ivan Levkivskyi) in branch 'master':
bpo-29638: Fix spurious refleaks after typing is imported (#469)
https://github.com/python/cpython/commit/7acffa23c9e7866e5b4b98b8f3c28b14ec1c688c


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29638] Spurious failures in test_collections in releak hunting mode after typing is imported

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
resolution:  -> fixed
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29638] Spurious failures in test_collections in releak hunting mode after typing is imported

[Ivan Levkivskyi]

Changes by Ivan Levkivskyi :


--
pull_requests: +395

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29638] Spurious failures in test_collections in releak hunting mode after typing is imported

[Ivan Levkivskyi]

Changes by Ivan Levkivskyi :


--
pull_requests: +394

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29638] Spurious failures in test_collections in releak hunting mode after typing is imported

[Ivan Levkivskyi]

Ivan Levkivskyi added the comment:

> Seems this has fixed issue25744.

This is interesting, if I remember correctly the relevant typing classes were 
added only recently. I will take a look at how to back-port this (probably this 
will require some code changes)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29638] Spurious failures in test_collections in releak hunting mode after typing is imported

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Thank you Ivan! Seems this has fixed issue25744. Don't you mind to backport 
your changes to 3.6 and 3.5?

--
versions: +Python 3.5, Python 3.6, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29638] Spurious failures in test_collections in releak hunting mode after typing is imported

[Ivan Levkivskyi]

Ivan Levkivskyi added the comment:

> If you just want to add a workaround in dash_R_cleanup, I think it would be 
> better to generate the list of all abstract classes and add three typing 
> classes to it.

Yes, I just updated the PR.

I have found something else unrelated to typing, this test (in exactly this 
order):

./python -m test -R 5:5 test_abc test_functools

fails with a reproducible pattern:

test_functools leaked [0, 3, 1, 0, 0] memory blocks, sum=4

If you are happy with the PR now, we could open two separate issues for 
possible bug in inspect.isabstract and refleak in test_functools after 
test_abc. (the latter however might be potentially somehow related to abc 
caches)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29638] Spurious failures in test_collections in releak hunting mode after typing is imported

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

I just wonder is there any other negative consequences? inspect.isabstract() 
returning False for abstract class looks a bug to me.

If you just want to add a workaround in dash_R_cleanup, I think it would be 
better to generate the list of all abstract classes and add three typing 
classes to it.

clear_caches() is called before running every test, but ABC caches are cleared 
not so often.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29638] Spurious failures in test_collections in releak hunting mode after typing is imported

[Ivan Levkivskyi]

Ivan Levkivskyi added the comment:

> What if explicitly set __abstractmethods__ = True for these types?

Unfortunately this does not help. I think this is because dash_R_cleanup only 
clears caches for classes in collections.abc.__all__ and their immediate 
.__subclasses__(). Making this recursive (i.e. also clearing caches of 
__subclasses__() of __subclasses__() etc) will probably fix the problem 
(provided we also add __abstractmethods__ = True). However, this looks a bit 
too complex for me. I would rather add few more typing._cleanups (anyway this 
failure only happens with typing ABCs)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29638] Spurious failures in test_collections in releak hunting mode after typing is imported

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

What if explicitly set __abstractmethods__ = True for these types?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29638] Spurious failures in test_collections in releak hunting mode after typing is imported

[Ivan Levkivskyi]

Ivan Levkivskyi added the comment:

> Are typing.ChainMap and others actually abstract classes?

They are abstract classes in the sense that they are instances of abc.ABCMeta. 
However, for some reasons inspect checks __flags__ attribute. The latter 
probably reflects the fact that Deque etc. do not have abstract methods (they 
still need to be instances of abc.ABCMeta because they need to be generic).

I don't think that we need to change behaviour of inspect because it could be 
backward incompatible. I would either make changes to refleak or typing (adding 
few more cleanups in either place).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29638] Spurious failures in test_collections in releak hunting mode after typing is imported

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

> So maybe we just need to add all ABC caches to typing._cleanups (just in 
> case).

I don't know. Sorry, I didn't remember exactly reasons why I didn't move the 
clearing of all ABC caches into clear_caches(). Perhaps I suspected that this 
can have unwanted consequences or decrease performance too much.

Are typing.ChainMap and others actually abstract classes? If yes, then perhaps 
there is something wrong with inspect.isabstract(). If no, then why ABC caches 
for these classes exist and non-empty?

--
nosy: +haypo, rhettinger, stutzbach

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29638] Spurious failures in test_collections in releak hunting mode after typing is imported

[Ivan Levkivskyi]

Ivan Levkivskyi added the comment:

> why not add corresponding clearing methods obj._abc_cache.clear and 
> obj._abc_negative_cache.clear to typing._cleanups ?

OK, this is another possible solution. I didn't think about this, because now 
typing._cleanups only clear generic caches (not ABC caches).

> Why there are problems only with ABC caches of these three types?

Normally, ABC caches are cleared in dash_R_cleanup, but I think that the 
problem is that inspect.isabstract returns False for these three types (unlike 
other types in typing module), see line 147 in refleak.py:

if not isabstract(abc):
continue

Actually now I think that the code in dash_R_cleanup might not clear some other 
caches also. So maybe we just need to add all ABC caches to typing._cleanups 
(just in case). What do you think, Serhiy?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29638] Spurious failures in test_collections in releak hunting mode after typing is imported

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

If there are problems with ABC caches of typing.ChainMap, typing.Counter, and 
typing.DefaultDict, why not add corresponding clearing methods 
obj._abc_cache.clear and obj._abc_negative_cache.clear to typing._cleanups ?

Why there are problems only with ABC caches of these three types?

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29638] Spurious failures in test_collections in releak hunting mode after typing is imported

[Ivan Levkivskyi]

Changes by Ivan Levkivskyi :


--
pull_requests: +387

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29638] Spurious failures in test_collections in releak hunting mode after typing is imported

[Ivan Levkivskyi]

New submission from Ivan Levkivskyi:

This command:

./python -c 'import runpy, typing; runpy.run_module("test")' -R 5:5 
test_collections

Sometimes gives spurious failures like this:

test_collections leaked [0, 0, 3, -3, 3] memory blocks, sum=3

I think this is because ABC caches of typing.ChainMap, typing.Counter, and 
typing.DefaultDict are not cleared by refleak.py/dash_R_cleanup (presumably 
because inspect.isabstract returns False on those)

Adding a manual clean-up of these cashes to cleanup_cashes() fixes the "leak".

--
assignee: levkivskyi
components: Tests
messages: 288495
nosy: gvanrossum, levkivskyi
priority: normal
severity: normal
status: open
title: Spurious failures in test_collections in releak hunting mode after 
typing is imported
type: resource usage

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com