[issue26617] Assertion failed in gc with __del__ and weakref

[Ned Deily]

Change by Ned Deily :


--
Removed message: https://bugs.python.org/msg342107

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26617] Assertion failed in gc with __del__ and weakref

[Ned Deily]

Ned Deily  added the comment:


New changeset 34fae03cd6c9e304e02c571b3bf9e8df0cfe76be by larryhastings (Serhiy 
Storchaka) in branch '3.4':
[3.4] bpo-26617: Ensure gc tracking is off when invoking weakref callbacks. 
(#2695)
https://github.com/python/cpython/commit/34fae03cd6c9e304e02c571b3bf9e8df0cfe76be


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26617] Assertion failed in gc with __del__ and weakref

[Alexander Mohr]

Alexander Mohr added the comment:

so I just discovered that the object that has the zero refcount has the same 
tp_dealloc:

(gdb) print *FROM_GC(gc)->ob_type
$8 = {ob_base = {ob_base = {_ob_next = 0x7f80c8aafc88, _ob_prev = 
0x7f80c8aafd00, ob_refcnt = 7, ob_type = 0x7f80cd8c86e0 }, ob_size 
= 0}, tp_name = 0x7f80c8aa5c38 "_GatheringFuture", tp_basicsize = 104, 
tp_itemsize = 0,
  tp_dealloc = 0x7f80cd41bee7 , tp_print = 0x0, tp_getattr = 
0x0, tp_setattr = 0x0, tp_as_async = 0x556ba4342d58, tp_repr = 0x7f80c8870a9b 
, tp_as_number = 0x556ba4342d70, tp_as_sequence = 
0x556ba4342ea8,
  tp_as_mapping = 0x556ba4342e90, tp_hash = 0x7f80cd54c995 <_Py_HashPointer>, 
tp_call = 0x0, tp_str = 0x7f80cd423d05 , tp_getattro = 
0x7f80cd3fec8b , tp_setattro = 0x7f80cd3ff0b5 
,
  tp_as_buffer = 0x556ba4342ef8, tp_flags = 808449, tp_doc = 0x7f80c8cd7380 
"Helper for gather().\n\nThis overrides cancel() to cancel all the children 
and act more\nlike Task.cancel(), which doesn't immediately mark itself 
as\ncancelled.\n",
  tp_traverse = 0x7f80cd41baae , tp_clear = 0x7f80cd41bd5c 
, tp_richcompare = 0x7f80cd423d42 , 
tp_weaklistoffset = 96, tp_iter = 0x7f80c8871d4f ,
  tp_iternext = 0x7f80cd3fe6d6 <_PyObject_NextNotImplemented>, tp_methods = 
0x0, tp_members = 0x556ba4342f28, tp_getset = 0x0, tp_base = 0x7f80c8a7b3c0 
, tp_dict = 0x7f80c8aafc88, tp_descr_get = 0x0, tp_descr_set = 0x0, 
tp_dictoffset = 88,
  tp_init = 0x7f80cd431000 , tp_alloc = 0x7f80cd41b7f8 
, tp_new = 0x7f80cd41b9c6 , tp_free = 
0x7f80cd583d8b , tp_is_gc = 0x0, tp_bases = 0x7f80c8ab20c0, 
tp_mro = 0x7f80c8aafd00, 
  tp_cache = 0x0, tp_subclasses = 0x0, tp_weaklist = 0x7f80c8aae5d8, tp_del = 
0x0, tp_version_tag = 791, tp_finalize = 0x7f80c8870ddb }

This is for a GatheringFuture, something tells me perhaps there is more to this 
function that needs to be resolved?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26617] Assertion failed in gc with __del__ and weakref

[Alexander Mohr]

Alexander Mohr added the comment:

I'm tracking something very similar issue to this in bug: 
http://bugs.python.org/issue31061  Given its similarities, anyone have any 
ideas?  Based on the second callstack I'm starting to think this is an issue 
with defaultdict

--
nosy: +thehesiod

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26617] Assertion failed in gc with __del__ and weakref

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
pull_requests:  -953

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26617] Assertion failed in gc with __del__ and weakref

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Seems this isn't reproducible with 3.3.

--
priority: release blocker -> normal
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26617] Assertion failed in gc with __del__ and weakref

[Larry Hastings]

Larry Hastings added the comment:

Sorry for the delay in merging your PR.  I believe Ned is planning to tag the 
next 3.3 rc1 tomorrow (Sunday).  If you see this, and you have time, it'd be 
great if you could make a PR for the 3.3 backport.

Note that 3.3 is probably *not* getting converted to use "blurb".  So if you do 
make the backport, please just edit Misc/NEWS directly.

Thanks!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26617] Assertion failed in gc with __del__ and weakref

[Larry Hastings]

Larry Hastings added the comment:


New changeset 34fae03cd6c9e304e02c571b3bf9e8df0cfe76be by larryhastings (Serhiy 
Storchaka) in branch '3.4':
[3.4] bpo-26617: Ensure gc tracking is off when invoking weakref callbacks. 
(#2695)
https://github.com/python/cpython/commit/34fae03cd6c9e304e02c571b3bf9e8df0cfe76be


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26617] Assertion failed in gc with __del__ and weakref

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

I wait merging PR 2695 for cherry-picking it to 3.3.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26617] Assertion failed in gc with __del__ and weakref

[Ned Deily]

Ned Deily added the comment:

If it's also reproducible with 3.3, I'd take a 3.3 PR for it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26617] Assertion failed in gc with __del__ and weakref

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
pull_requests: +2759

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26617] Assertion failed in gc with __del__ and weakref

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

I'll do this.

--
nosy: +serhiy.storchaka
status: closed -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26617] Assertion failed in gc with __del__ and weakref

[Larry Hastings]

Larry Hastings added the comment:

I'll accept this as a backport for 3.4.  I'm convinced that, although the 
sample programs don't crash in release builds, the behavior is wrong and could 
be exploited.

Will someone have time to create the PR in the next day or two?  (It's not that 
it's necessary, but it seems that Victor and Serhiy are very quick, which is 
nice.  And adding it sooner will make it more comfortable to tag and release 
3.4.7rc1 next weekend, which I think I'm going to do.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26617] Assertion failed in gc with __del__ and weakref

[Donald Stufft]

Changes by Donald Stufft :


--
pull_requests: +953

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26617] Assertion failed in gc with __del__ and weakref

[STINNER Victor]

STINNER Victor added the comment:

> You should ask for a downstream backport.

I created the issue: https://bugzilla.redhat.com/show_bug.cgi?id=1384957

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26617] Assertion failed in gc with __del__ and weakref

[STINNER Victor]

STINNER Victor added the comment:

> Would be nice if 3.4 could also be patched, not just 3.5 and 3.6, since 
> python in EPEL currently is python34-3.4.3-7.el7.x86_64. The patch can be 
> applied without conflict and resolves some serious cases of segfaults.

Sorry but Python 3.4 is not more supported upstream:
https://docs.python.org/devguide/#status-of-python-branches

You should ask for a downstream backport.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26617] Assertion failed in gc with __del__ and weakref

[Frens Jan Rumph]

Frens Jan Rumph added the comment:

Would be nice if 3.4 could also be patched, not just 3.5 and 3.6, since python 
in EPEL currently is python34-3.4.3-7.el7.x86_64. The patch can be applied 
without conflict and resolves some serious cases of segfaults.

--
nosy: +Frens Jan Rumph

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26617] Assertion failed in gc with __del__ and weakref

[Roundup Robot]

Roundup Robot added the comment:

New changeset c9b7272e2553 by Benjamin Peterson in branch '3.5':
ensure gc tracking is off when invoking weakref callbacks (closes #26617)
https://hg.python.org/cpython/rev/c9b7272e2553

New changeset 520cb70ecb90 by Benjamin Peterson in branch '3.6':
merge 3.5 (#26617)
https://hg.python.org/cpython/rev/520cb70ecb90

New changeset c1d0df056c19 by Benjamin Peterson in branch 'default':
merge 3.6 (#26617)
https://hg.python.org/cpython/rev/c1d0df056c19

--
nosy: +python-dev
resolution:  -> fixed
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26617] Assertion failed in gc with __del__ and weakref

[JIahua Guo]

JIahua Guo added the comment:

Hi asvetlov, debug mode of python interpreter should be enabled to reproduce 
this bug, cause assertion is disabled in release mode.
(https://hg.python.org/cpython/file/104259/Modules/gcmodule.c#l365)

One environment that can reproduce this bug:

$ cat /etc/issue
Ubuntu 14.04.3 LTS \n \l

$ uname -r
3.13.0-32-generic

$ uname -m
x86_64

$ dpkg -s python3.4-dbg | grep Version
Version: 3.4.3-1ubuntu1~14.04.4

$ python3.4-dbg crash.py
python3.4-dbg: ../Modules/gcmodule.c:364: update_refs: Assertion 
`((gc)->gc.gc_refs >> (1)) != 0' failed.
Aborted

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26617] Assertion failed in gc with __del__ and weakref

[Andrew Svetlov]

Andrew Svetlov added the comment:

I was unable to reproduce a crash but the patch looks straightforward and 
obvious.
After applying test suite for aiohttp still works without problems at least.

--
nosy: +asvetlov

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26617] Assertion failed in gc with __del__ and weakref

[STINNER Victor]

STINNER Victor added the comment:

I don't think that the crash is a release blocker, but I just sent an email to 
python-dev to ask for reviews.

--
nosy: +larry, ned.deily
priority: normal -> release blocker

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26617] Assertion failed in gc with __del__ and weakref

[STINNER Victor]

Changes by STINNER Victor :


--
versions: +Python 3.6, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26617] Assertion failed in gc with __del__ and weakref

[Paulus Schoutsen]

Changes by Paulus Schoutsen :


--
nosy: +balloob

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26617] Assertion failed in gc with __del__ and weakref

[Guido van Rossum]

Guido van Rossum added the comment:

Ben Bangert reported to me that this crash caused instabilities in an app using 
asyncio (https://github.com/home-assistant/home-assistant/issues/3453). This 
hack made his crashes go away:
https://github.com/home-assistant/home-assistant/commit/922dbba8814b81b69471dc4f1bf5c5a3b2bfe4ed

What are the chances of getting the crash fixed in 3.5.3 and 3.6b2?

--
nosy: +gvanrossum

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26617] Assertion failed in gc with __del__ and weakref

[Guo, Jiahua]

Changes by Guo, Jiahua :


Added file: http://bugs.python.org/file42892/patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26617] Assertion failed in gc with __del__ and weakref

[STINNER Victor]

Changes by STINNER Victor :


--
nosy: +tim.peters

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26617] Assertion failed in gc with __del__ and weakref

[STINNER Victor]

STINNER Victor added the comment:

Oh, I confirm that the bug occurs on Python 3.5.1+ (3.5 branch) and Python 3.6 
(default branch).

Call stack: _Py_Dealloc() => subtype_dealloc() => PyObject_ClearWeakRefs() => 
handle_callback() of weakref => ... => gc_collect => ... => assertion error

Full gdb tracebacks:

python: Modules/gcmodule.c:364: update_refs: Assertion `((gc)->gc.gc_refs >> 
(1)) != 0' failed.

Program received signal SIGABRT, Aborted.
0x77129a98 in raise () from /lib64/libc.so.6

(gdb) where
#0  0x77129a98 in raise () from /lib64/libc.so.6
#1  0x7712b69a in abort () from /lib64/libc.so.6
#2  0x7717 in __assert_fail_base () from /lib64/libc.so.6
#3  0x771222d2 in __assert_fail () from /lib64/libc.so.6
#4  0x00441b74 in update_refs (containers=0x939c60 ) at 
Modules/gcmodule.c:364
#5  0x00442cb1 in collect (generation=2, n_collected=0x7fff87f0, 
n_uncollectable=0x7fff87e8, nofail=0) at Modules/gcmodule.c:950
#6  0x004432dd in collect_with_callback (generation=2) at 
Modules/gcmodule.c:1119
#7  0x004434cb in gc_collect (self=, 
args=(), kws=0x0) at Modules/gcmodule.c:1213
#8  0x004d5731 in PyCFunction_Call (func=, args=(), kwds=0x0)
at Objects/methodobject.c:98
#9  0x005d0fd3 in call_function (pp_stack=0x7fff8a18, oparg=0) at 
Python/ceval.c:4732
#10 0x005c8cc0 in PyEval_EvalFrameEx (f=Frame 0x77eaa858, for file 
crash.py, line 7, in callback (o=), 
throwflag=0) at Python/ceval.c:3207
#11 0x005ce4ec in _PyEval_EvalCodeWithName (_co=, 
globals={'__package__': None, '__name__': '__main__', 'gc': , 'weakref': , 
'__builtins__': , 'C': , '__doc__': None, '__spec__': None, 'main': , 'callback': , '__loader__': 
, 
'__file__': 'crash.py', '__cached__': None}, locals=0x0, args=0x705d5b78, 
argcount=1, kws=0x0, kwcount=0, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, 
name=0x0, qualname=0x0)
at Python/ceval.c:4045
#12 0x005ce61e in PyEval_EvalCodeEx (_co=, 
globals={'__package__': None, '__name__': '__main__', 'gc': , 'weakref': , 
'__builtins__': , 'C': , '__doc__': None, '__spec__': None, 'main': , 'callback': , '__loader__': 
, 
'__file__': 'crash.py', '__cached__': None}, locals=0x0, args=0x705d5b78, 
argcount=1, kws=0x0, kwcount=0, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0) 
at Python/ceval.c:4066
#13 0x00688276 in function_call (func=, arg=(,), kw=0x0)
at Objects/funcobject.c:604
#14 0x0046c4f3 in PyObject_Call (func=, arg=(,), kw=0x0)
at Objects/abstract.c:2170
#15 0x0046d61d in PyObject_CallFunctionObjArgs (callable=) at Objects/abstract.c:2450
#16 0x00576063 in handle_callback (ref=0x705eb658, 
callback=) at Objects/weakrefobject.c:868
#17 0x00576257 in PyObject_ClearWeakRefs (object=) at Objects/weakrefobject.c:913
#18 0x004f3bee in subtype_dealloc (self=) 
at Objects/typeobject.c:1162
#19 0x004dd7d6 in _Py_Dealloc (op=) at 
Objects/object.c:1783
(...)

(gdb) py-bt
Traceback (most recent call first):
  Garbage-collecting
  
  File "crash.py", line 7, in callback
gc.collect()
  File "crash.py", line 17, in main
del c
  File "crash.py", line 20, in 
main()

--
Added file: http://bugs.python.org/file42254/crash.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26617] Assertion failed in gc with __del__ and weakref

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
nosy: +fdrake, haypo, pitrou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26617] Assertion failed in gc with __del__ and weakref

[Xiang Zhang]

Changes by Xiang Zhang :


--
nosy: +xiang.zhang

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26617] Assertion failed in gc with __del__ and weakref

[Guo Jiahua]

Changes by Guo Jiahua :


--
nosy: +Guo Jiahua

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26617] Assertion failed in gc with __del__ and weakref

[Yang Ke]

Yang Ke added the comment:

The crash may be raised on both Windows OS and Linux. We got crashed under 
Ubuntu 14.04, while a similar issue was found 
https://bugs.python.org/issue23187#msg238139 last year under Windows OS.

We actually raise such bug working on a program with asyncio and 
multithread(use asyncio.BaseEventLoop.run_in_executor with its default 
threadpool).
The asyncio.Task class creates a 
weakref.WeakSet(https://hg.python.org/cpython/file/3.5/Lib/asyncio/tasks.py#l38)
 containing alive tasks. WeakSet is implemented as a list containing 
weakref.ref with CALLBACK 
function(https://hg.python.org/cpython/file/3.5/Lib/_weakrefset.py#l84). And 
that may lead to the situation described in sample crash code:
Once a asyncio.Task.__del__ function is called, and callback of the weakref is 
called, then the interpreter switches to other thread, which may trigger 
gc.collect(the thread allocate a huge number of objects, for example), and 
leaves the task object in gc_list with refcount==0.

--
nosy: +Yang Ke -Guo. Jiahua

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26617] Assertion failed in gc with __del__ and weakref

[Guo, Jiahua]

New submission from Guo, Jiahua:

$ python3.5-dbg crash.py
python3.5-dbg: ../Modules/gcmodule.c:364: update_refs: Assertion 
`((gc)->gc.gc_refs >> (1)) != 0' failed.


$ python3.5-dbg crash-multithread.py
python3.5-dbg: ../Modules/gcmodule.c:364: update_refs: Assertion 
`((gc)->gc.gc_refs >> (1)) != 0' failed.


 crash.py ==
import weakref
import gc


def callback(o):
gc.collect()


class C:
def __del__(self):
pass

def main():
c = C()
cref = weakref.ref(c, callback)
del c


main()


 crash-multithread.py ==
import threading
import weakref
import gc


ei = threading.Event()
eo = threading.Event()


def gc_worker():
ei.wait()
gc.collect()
eo.set()


def callback(o):
ei.set()
eo.wait()


class C:
def __del__(self):
pass


def main():
t = threading.Thread(target=gc_worker)
t.start()
c = C()
cref = weakref.ref(c, callback)
del c
t.join()


main()

--
components: Interpreter Core
messages: 262242
nosy: Guo, Jiahua
priority: normal
severity: normal
status: open
title: Assertion failed in gc with __del__ and weakref
type: crash
versions: Python 3.4, Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com