Re: [vdsm] suggested patch for python-pthreading
- Original Message - > From: "ybronhei" > To: "VDSM Project Development" , "Nir > Soffer" > Sent: Tuesday, February 25, 2014 8:59:06 AM > Subject: Fwd: Fwd: [vdsm] suggested patch for python-pthreading > > back to the topic > I would like to continue pushing the fix in. > > Nir, iirc you suggested a patch to validate that > pthreading.monkey_patch() is being called before using the native thread > and pthreading module, right? > any comments about that part? Yes, since we must import thread and monkey patch it before importing threading. As soon as user imported thread, it is too later because some lockes are created during the import. > > > > ---- Original Message > Subject: Fwd: [vdsm] suggested patch for python-pthreading > Date: Tue, 18 Feb 2014 09:32:59 -0500 (EST) > From: Nir Soffer > To: Yaniv Bronheim > > > > - Forwarded Message - > > From: "Yaniv Bronheim" > > To: "Nir Soffer" > > Cc: "Dan Kenigsberg" , "VDSM Project Development" > > > > Sent: Wednesday, February 5, 2014 6:41:28 PM > > Subject: Re: [vdsm] suggested patch for python-pthreading > > > > - Original Message ----- > > > From: "Nir Soffer" > > > To: "Dan Kenigsberg" > > > Cc: "Yaniv Bronheim" , "VDSM Project Development" > > > > > > Sent: Tuesday, February 4, 2014 4:39:55 PM > > > Subject: Re: [vdsm] suggested patch for python-pthreading > > > > > > - Original Message - > > > > From: "Dan Kenigsberg" > > > > To: "Nir Soffer" > > > > Cc: "Yaniv Bronheim" , "VDSM Project Development" > > > > > > > > Sent: Tuesday, February 4, 2014 3:51:02 PM > > > > Subject: Re: [vdsm] suggested patch for python-pthreading > > > > > > > > On Tue, Feb 04, 2014 at 05:14:51AM -0500, Nir Soffer wrote: > > > > > - Original Message - > > > > > > From: "Yaniv Bronheim" > > > > > > To: "Nir Soffer" > > > > > > Cc: "VDSM Project Development" > > > > > > Sent: Tuesday, February 4, 2014 11:53:52 AM > > > > > > Subject: Re: [vdsm] suggested patch for python-pthreading > > > > > > > > > > > > > > > > > > > > > > > > - Original Message - > > > > > > > From: "Nir Soffer" > > > > > > > To: "Yaniv Bronheim" > > > > > > > Cc: "VDSM Project Development" > > > > > > > > > > > > > > Sent: Tuesday, February 4, 2014 11:39:56 AM > > > > > > > Subject: Re: [vdsm] suggested patch for python-pthreading > > > > > > > > > > > > > > - Original Message - > > > > > > > > From: "Yaniv Bronheim" > > > > > > > > To: "VDSM Project Development" > > > > > > > > > > > > > > > > Sent: Tuesday, February 4, 2014 11:04:37 AM > > > > > > > > Subject: [vdsm] suggested patch for python-pthreading > > > > > > > > > > > > > > > > according to coredumps we found in the scope of the bug [1] we > > > > > > > > opened > > > > > > > > [2] > > > > > > > > that suggested to override python's implementation of > > > > > > > > thread.allocate_lock > > > > > > > > in each coredump we saw few threads stuck with the bt: > > > > > > > > > > > > > > > > #16 0x7fcb69288c93 in PyEval_CallObjectWithKeywords > > > > > > > > (func=0x2527820, > > > > > > > > arg=0x7fcb6972f050, kw=) at > > > > > > > > Python/ceval.c:3663 > > > > > > > > #17 0x7fcb692ba7ba in t_bootstrap (boot_raw=0x250a820) at > > > > > > > > Modules/threadmodule.c:428 > > > > > > > > #18 0x7fcb68fa3851 in start_thread (arg=0x7fcb1bfff700) at > > > > > > > > pthread_create.c:301 > > > > > > > > #19 0x7fcb6866694d in clone () at > > > > > > > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 > > > > > > > &
Re: [vdsm] suggested patch for python-pthreading
- Original Message - > From: "Nir Soffer" > To: "Dan Kenigsberg" > Cc: "VDSM Project Development" > Sent: Wednesday, February 5, 2014 10:05:22 PM > Subject: Re: [vdsm] suggested patch for python-pthreading > > - Original Message - > > From: "Dan Kenigsberg" > > To: "Yaniv Bronheim" > > Cc: "Nir Soffer" , "VDSM Project Development" > > > > Sent: Wednesday, February 5, 2014 9:23:08 PM > > Subject: Re: [vdsm] suggested patch for python-pthreading > > > > On Wed, Feb 05, 2014 at 11:41:28AM -0500, Yaniv Bronheim wrote: > > > - Original Message - > > > > From: "Nir Soffer" > > > > To: "Dan Kenigsberg" > > > > Cc: "Yaniv Bronheim" , "VDSM Project Development" > > > > > > > > Sent: Tuesday, February 4, 2014 4:39:55 PM > > > > Subject: Re: [vdsm] suggested patch for python-pthreading > > > > > > > > - Original Message - > > > > > From: "Dan Kenigsberg" > > > > > To: "Nir Soffer" > > > > > Cc: "Yaniv Bronheim" , "VDSM Project > > > > > Development" > > > > > > > > > > Sent: Tuesday, February 4, 2014 3:51:02 PM > > > > > Subject: Re: [vdsm] suggested patch for python-pthreading > > > > > > > > > > On Tue, Feb 04, 2014 at 05:14:51AM -0500, Nir Soffer wrote: > > > > > > - Original Message - > > > > > > > From: "Yaniv Bronheim" > > > > > > > To: "Nir Soffer" > > > > > > > Cc: "VDSM Project Development" > > > > > > > > > > > > > > Sent: Tuesday, February 4, 2014 11:53:52 AM > > > > > > > Subject: Re: [vdsm] suggested patch for python-pthreading > > > > > > > > > > > > > > > > > > > > > > > > > > > > - Original Message - > > > > > > > > From: "Nir Soffer" > > > > > > > > To: "Yaniv Bronheim" > > > > > > > > Cc: "VDSM Project Development" > > > > > > > > > > > > > > > > Sent: Tuesday, February 4, 2014 11:39:56 AM > > > > > > > > Subject: Re: [vdsm] suggested patch for python-pthreading > > > > > > > > > > > > > > > > - Original Message - > > > > > > > > > From: "Yaniv Bronheim" > > > > > > > > > To: "VDSM Project Development" > > > > > > > > > > > > > > > > > > Sent: Tuesday, February 4, 2014 11:04:37 AM > > > > > > > > > Subject: [vdsm] suggested patch for python-pthreading > > > > > > > > > > > > > > > > > > according to coredumps we found in the scope of the bug [1] > > > > > > > > > we > > > > > > > > > opened > > > > > > > > > [2] > > > > > > > > > that suggested to override python's implementation of > > > > > > > > > thread.allocate_lock > > > > > > > > > in each coredump we saw few threads stuck with the bt: > > > > > > > > > > > > > > > > > > #16 0x7fcb69288c93 in PyEval_CallObjectWithKeywords > > > > > > > > > (func=0x2527820, > > > > > > > > > arg=0x7fcb6972f050, kw=) at > > > > > > > > > Python/ceval.c:3663 > > > > > > > > > #17 0x7fcb692ba7ba in t_bootstrap (boot_raw=0x250a820) at > > > > > > > > > Modules/threadmodule.c:428 > > > > > > > > > #18 0x7fcb68fa3851 in start_thread (arg=0x7fcb1bfff700) > > > > > > > > > at > > > > > > > > > pthread_create.c:301 > > > > > > > > > #19 0x7fcb6866694d in clone () at > > > > > > > > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 > > > > > > > > > > > > > > > > > > in pystack the threads were stuck in > > > > > > > > > /usr/lib64/python2.6/threading.py > > > > > &
Re: [vdsm] suggested patch for python-pthreading
- Original Message - > From: "Dan Kenigsberg" > To: "Yaniv Bronheim" > Cc: "Nir Soffer" , "VDSM Project Development" > > Sent: Wednesday, February 5, 2014 9:23:08 PM > Subject: Re: [vdsm] suggested patch for python-pthreading > > On Wed, Feb 05, 2014 at 11:41:28AM -0500, Yaniv Bronheim wrote: > > - Original Message - > > > From: "Nir Soffer" > > > To: "Dan Kenigsberg" > > > Cc: "Yaniv Bronheim" , "VDSM Project Development" > > > > > > Sent: Tuesday, February 4, 2014 4:39:55 PM > > > Subject: Re: [vdsm] suggested patch for python-pthreading > > > > > > - Original Message - > > > > From: "Dan Kenigsberg" > > > > To: "Nir Soffer" > > > > Cc: "Yaniv Bronheim" , "VDSM Project Development" > > > > > > > > Sent: Tuesday, February 4, 2014 3:51:02 PM > > > > Subject: Re: [vdsm] suggested patch for python-pthreading > > > > > > > > On Tue, Feb 04, 2014 at 05:14:51AM -0500, Nir Soffer wrote: > > > > > - Original Message - > > > > > > From: "Yaniv Bronheim" > > > > > > To: "Nir Soffer" > > > > > > Cc: "VDSM Project Development" > > > > > > Sent: Tuesday, February 4, 2014 11:53:52 AM > > > > > > Subject: Re: [vdsm] suggested patch for python-pthreading > > > > > > > > > > > > > > > > > > > > > > > > - Original Message - > > > > > > > From: "Nir Soffer" > > > > > > > To: "Yaniv Bronheim" > > > > > > > Cc: "VDSM Project Development" > > > > > > > > > > > > > > Sent: Tuesday, February 4, 2014 11:39:56 AM > > > > > > > Subject: Re: [vdsm] suggested patch for python-pthreading > > > > > > > > > > > > > > - Original Message - > > > > > > > > From: "Yaniv Bronheim" > > > > > > > > To: "VDSM Project Development" > > > > > > > > > > > > > > > > Sent: Tuesday, February 4, 2014 11:04:37 AM > > > > > > > > Subject: [vdsm] suggested patch for python-pthreading > > > > > > > > > > > > > > > > according to coredumps we found in the scope of the bug [1] we > > > > > > > > opened > > > > > > > > [2] > > > > > > > > that suggested to override python's implementation of > > > > > > > > thread.allocate_lock > > > > > > > > in each coredump we saw few threads stuck with the bt: > > > > > > > > > > > > > > > > #16 0x7fcb69288c93 in PyEval_CallObjectWithKeywords > > > > > > > > (func=0x2527820, > > > > > > > > arg=0x7fcb6972f050, kw=) at > > > > > > > > Python/ceval.c:3663 > > > > > > > > #17 0x7fcb692ba7ba in t_bootstrap (boot_raw=0x250a820) at > > > > > > > > Modules/threadmodule.c:428 > > > > > > > > #18 0x7fcb68fa3851 in start_thread (arg=0x7fcb1bfff700) at > > > > > > > > pthread_create.c:301 > > > > > > > > #19 0x7fcb6866694d in clone () at > > > > > > > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 > > > > > > > > > > > > > > > > in pystack the threads were stuck in > > > > > > > > /usr/lib64/python2.6/threading.py > > > > > > > > (513): __bootstrap_inner > > > > > > > > > > > > > > > > in bootstrap_inner we use thread.allocate_lock which > > > > > > > > python-pthreading > > > > > > > > does > > > > > > > > not override. > > > > > > > > > > > > > > > > we suggest the following commit: > > > > > > > > > > > > > > > > From 9d89e9be1a379b3d93b23dd54a381b9ca0973ebc Mon Sep 17 > > > > > > > > 00:00:00 > > > > > > > > 2001 > > > > > > > > From: Yaniv Bronhaim > >
Re: [vdsm] suggested patch for python-pthreading
On Wed, Feb 05, 2014 at 11:41:28AM -0500, Yaniv Bronheim wrote: > - Original Message - > > From: "Nir Soffer" > > To: "Dan Kenigsberg" > > Cc: "Yaniv Bronheim" , "VDSM Project Development" > > > > Sent: Tuesday, February 4, 2014 4:39:55 PM > > Subject: Re: [vdsm] suggested patch for python-pthreading > > > > - Original Message - > > > From: "Dan Kenigsberg" > > > To: "Nir Soffer" > > > Cc: "Yaniv Bronheim" , "VDSM Project Development" > > > > > > Sent: Tuesday, February 4, 2014 3:51:02 PM > > > Subject: Re: [vdsm] suggested patch for python-pthreading > > > > > > On Tue, Feb 04, 2014 at 05:14:51AM -0500, Nir Soffer wrote: > > > > ----- Original Message - > > > > > From: "Yaniv Bronheim" > > > > > To: "Nir Soffer" > > > > > Cc: "VDSM Project Development" > > > > > Sent: Tuesday, February 4, 2014 11:53:52 AM > > > > > Subject: Re: [vdsm] suggested patch for python-pthreading > > > > > > > > > > > > > > > > > > > > - Original Message - > > > > > > From: "Nir Soffer" > > > > > > To: "Yaniv Bronheim" > > > > > > Cc: "VDSM Project Development" > > > > > > Sent: Tuesday, February 4, 2014 11:39:56 AM > > > > > > Subject: Re: [vdsm] suggested patch for python-pthreading > > > > > > > > > > > > - Original Message - > > > > > > > From: "Yaniv Bronheim" > > > > > > > To: "VDSM Project Development" > > > > > > > Sent: Tuesday, February 4, 2014 11:04:37 AM > > > > > > > Subject: [vdsm] suggested patch for python-pthreading > > > > > > > > > > > > > > according to coredumps we found in the scope of the bug [1] we > > > > > > > opened > > > > > > > [2] > > > > > > > that suggested to override python's implementation of > > > > > > > thread.allocate_lock > > > > > > > in each coredump we saw few threads stuck with the bt: > > > > > > > > > > > > > > #16 0x7fcb69288c93 in PyEval_CallObjectWithKeywords > > > > > > > (func=0x2527820, > > > > > > > arg=0x7fcb6972f050, kw=) at > > > > > > > Python/ceval.c:3663 > > > > > > > #17 0x7fcb692ba7ba in t_bootstrap (boot_raw=0x250a820) at > > > > > > > Modules/threadmodule.c:428 > > > > > > > #18 0x7fcb68fa3851 in start_thread (arg=0x7fcb1bfff700) at > > > > > > > pthread_create.c:301 > > > > > > > #19 0x7fcb6866694d in clone () at > > > > > > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 > > > > > > > > > > > > > > in pystack the threads were stuck in > > > > > > > /usr/lib64/python2.6/threading.py > > > > > > > (513): __bootstrap_inner > > > > > > > > > > > > > > in bootstrap_inner we use thread.allocate_lock which > > > > > > > python-pthreading > > > > > > > does > > > > > > > not override. > > > > > > > > > > > > > > we suggest the following commit: > > > > > > > > > > > > > > From 9d89e9be1a379b3d93b23dd54a381b9ca0973ebc Mon Sep 17 00:00:00 > > > > > > > 2001 > > > > > > > From: Yaniv Bronhaim > > > > > > > Date: Mon, 3 Feb 2014 19:24:30 +0200 > > > > > > > Subject: [PATCH] Mocking thread.allocate_lock with Lock imp > > > > > > > > > > > > > > Signed-off-by: Yaniv Bronhaim > > > > > > > --- > > > > > > > pthreading.py | 4 > > > > > > > 1 file changed, 4 insertions(+) > > > > > > > > > > > > > > diff --git a/pthreading.py b/pthreading.py > > > > > > > index 916ca7f..96df42c 100644 > > > > > > > --- a/pthreading.py > > > > > > > +++ b/pthreading.py > > > > > > > @@ -132,6 +132,10 @@ def monkey_patch(): > &
Re: [vdsm] suggested patch for python-pthreading
- Original Message - > From: "Nir Soffer" > To: "Dan Kenigsberg" > Cc: "Yaniv Bronheim" , "VDSM Project Development" > > Sent: Tuesday, February 4, 2014 4:39:55 PM > Subject: Re: [vdsm] suggested patch for python-pthreading > > - Original Message - > > From: "Dan Kenigsberg" > > To: "Nir Soffer" > > Cc: "Yaniv Bronheim" , "VDSM Project Development" > > > > Sent: Tuesday, February 4, 2014 3:51:02 PM > > Subject: Re: [vdsm] suggested patch for python-pthreading > > > > On Tue, Feb 04, 2014 at 05:14:51AM -0500, Nir Soffer wrote: > > > - Original Message - > > > > From: "Yaniv Bronheim" > > > > To: "Nir Soffer" > > > > Cc: "VDSM Project Development" > > > > Sent: Tuesday, February 4, 2014 11:53:52 AM > > > > Subject: Re: [vdsm] suggested patch for python-pthreading > > > > > > > > > > > > > > > > - Original Message - > > > > > From: "Nir Soffer" > > > > > To: "Yaniv Bronheim" > > > > > Cc: "VDSM Project Development" > > > > > Sent: Tuesday, February 4, 2014 11:39:56 AM > > > > > Subject: Re: [vdsm] suggested patch for python-pthreading > > > > > > > > > > - Original Message - > > > > > > From: "Yaniv Bronheim" > > > > > > To: "VDSM Project Development" > > > > > > Sent: Tuesday, February 4, 2014 11:04:37 AM > > > > > > Subject: [vdsm] suggested patch for python-pthreading > > > > > > > > > > > > according to coredumps we found in the scope of the bug [1] we > > > > > > opened > > > > > > [2] > > > > > > that suggested to override python's implementation of > > > > > > thread.allocate_lock > > > > > > in each coredump we saw few threads stuck with the bt: > > > > > > > > > > > > #16 0x7fcb69288c93 in PyEval_CallObjectWithKeywords > > > > > > (func=0x2527820, > > > > > > arg=0x7fcb6972f050, kw=) at > > > > > > Python/ceval.c:3663 > > > > > > #17 0x7fcb692ba7ba in t_bootstrap (boot_raw=0x250a820) at > > > > > > Modules/threadmodule.c:428 > > > > > > #18 0x7fcb68fa3851 in start_thread (arg=0x7fcb1bfff700) at > > > > > > pthread_create.c:301 > > > > > > #19 0x7fcb6866694d in clone () at > > > > > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 > > > > > > > > > > > > in pystack the threads were stuck in > > > > > > /usr/lib64/python2.6/threading.py > > > > > > (513): __bootstrap_inner > > > > > > > > > > > > in bootstrap_inner we use thread.allocate_lock which > > > > > > python-pthreading > > > > > > does > > > > > > not override. > > > > > > > > > > > > we suggest the following commit: > > > > > > > > > > > > From 9d89e9be1a379b3d93b23dd54a381b9ca0973ebc Mon Sep 17 00:00:00 > > > > > > 2001 > > > > > > From: Yaniv Bronhaim > > > > > > Date: Mon, 3 Feb 2014 19:24:30 +0200 > > > > > > Subject: [PATCH] Mocking thread.allocate_lock with Lock imp > > > > > > > > > > > > Signed-off-by: Yaniv Bronhaim > > > > > > --- > > > > > > pthreading.py | 4 > > > > > > 1 file changed, 4 insertions(+) > > > > > > > > > > > > diff --git a/pthreading.py b/pthreading.py > > > > > > index 916ca7f..96df42c 100644 > > > > > > --- a/pthreading.py > > > > > > +++ b/pthreading.py > > > > > > @@ -132,6 +132,10 @@ def monkey_patch(): > > > > > > Thus, Queue and SocketServer can easily enjoy them. > > > > > > """ > > > > > > > > > > > > +import thread > > > > > > + > > > > > > +thread.allocate_lock = Lock > > > > > > + > > > > > > import threading > > > > > > > > > > > > threading.Condition = Co
Re: [vdsm] suggested patch for python-pthreading
- Original Message - > From: "Dan Kenigsberg" > To: "Nir Soffer" > Cc: "Yaniv Bronheim" , "VDSM Project Development" > > Sent: Tuesday, February 4, 2014 3:51:02 PM > Subject: Re: [vdsm] suggested patch for python-pthreading > > On Tue, Feb 04, 2014 at 05:14:51AM -0500, Nir Soffer wrote: > > - Original Message - > > > From: "Yaniv Bronheim" > > > To: "Nir Soffer" > > > Cc: "VDSM Project Development" > > > Sent: Tuesday, February 4, 2014 11:53:52 AM > > > Subject: Re: [vdsm] suggested patch for python-pthreading > > > > > > > > > > > > - Original Message ----- > > > > From: "Nir Soffer" > > > > To: "Yaniv Bronheim" > > > > Cc: "VDSM Project Development" > > > > Sent: Tuesday, February 4, 2014 11:39:56 AM > > > > Subject: Re: [vdsm] suggested patch for python-pthreading > > > > > > > > - Original Message - > > > > > From: "Yaniv Bronheim" > > > > > To: "VDSM Project Development" > > > > > Sent: Tuesday, February 4, 2014 11:04:37 AM > > > > > Subject: [vdsm] suggested patch for python-pthreading > > > > > > > > > > according to coredumps we found in the scope of the bug [1] we opened > > > > > [2] > > > > > that suggested to override python's implementation of > > > > > thread.allocate_lock > > > > > in each coredump we saw few threads stuck with the bt: > > > > > > > > > > #16 0x7fcb69288c93 in PyEval_CallObjectWithKeywords > > > > > (func=0x2527820, > > > > > arg=0x7fcb6972f050, kw=) at Python/ceval.c:3663 > > > > > #17 0x7fcb692ba7ba in t_bootstrap (boot_raw=0x250a820) at > > > > > Modules/threadmodule.c:428 > > > > > #18 0x7fcb68fa3851 in start_thread (arg=0x7fcb1bfff700) at > > > > > pthread_create.c:301 > > > > > #19 0x7fcb6866694d in clone () at > > > > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 > > > > > > > > > > in pystack the threads were stuck in > > > > > /usr/lib64/python2.6/threading.py > > > > > (513): __bootstrap_inner > > > > > > > > > > in bootstrap_inner we use thread.allocate_lock which > > > > > python-pthreading > > > > > does > > > > > not override. > > > > > > > > > > we suggest the following commit: > > > > > > > > > > From 9d89e9be1a379b3d93b23dd54a381b9ca0973ebc Mon Sep 17 00:00:00 > > > > > 2001 > > > > > From: Yaniv Bronhaim > > > > > Date: Mon, 3 Feb 2014 19:24:30 +0200 > > > > > Subject: [PATCH] Mocking thread.allocate_lock with Lock imp > > > > > > > > > > Signed-off-by: Yaniv Bronhaim > > > > > --- > > > > > pthreading.py | 4 > > > > > 1 file changed, 4 insertions(+) > > > > > > > > > > diff --git a/pthreading.py b/pthreading.py > > > > > index 916ca7f..96df42c 100644 > > > > > --- a/pthreading.py > > > > > +++ b/pthreading.py > > > > > @@ -132,6 +132,10 @@ def monkey_patch(): > > > > > Thus, Queue and SocketServer can easily enjoy them. > > > > > """ > > > > > > > > > > +import thread > > > > > + > > > > > +thread.allocate_lock = Lock > > > > > + > > > > > import threading > > > > > > > > > > threading.Condition = Condition > > > > > -- > > > > > 1.8.3.1 > > > > > > > > > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1022036 > > > > > [2] https://bugzilla.redhat.com/show_bug.cgi?id=1060749 > > > > > > > > > > > > > Replacing allocate_lock in thread is correct. However, since threading > > > > copies > > > > thread.allocate_lock, and you don't control import order, you have to > > > > monkeypatch > > > > threading.allocate_lock as well. > > > > > > > > The full should be: > > > > > > > > import thread > > > > thread.allocate_lock = Lock > > &
Re: [vdsm] suggested patch for python-pthreading
On Tue, Feb 04, 2014 at 05:14:51AM -0500, Nir Soffer wrote: > - Original Message - > > From: "Yaniv Bronheim" > > To: "Nir Soffer" > > Cc: "VDSM Project Development" > > Sent: Tuesday, February 4, 2014 11:53:52 AM > > Subject: Re: [vdsm] suggested patch for python-pthreading > > > > > > > > - Original Message - > > > From: "Nir Soffer" > > > To: "Yaniv Bronheim" > > > Cc: "VDSM Project Development" > > > Sent: Tuesday, February 4, 2014 11:39:56 AM > > > Subject: Re: [vdsm] suggested patch for python-pthreading > > > > > > - Original Message - > > > > From: "Yaniv Bronheim" > > > > To: "VDSM Project Development" > > > > Sent: Tuesday, February 4, 2014 11:04:37 AM > > > > Subject: [vdsm] suggested patch for python-pthreading > > > > > > > > according to coredumps we found in the scope of the bug [1] we opened > > > > [2] > > > > that suggested to override python's implementation of > > > > thread.allocate_lock > > > > in each coredump we saw few threads stuck with the bt: > > > > > > > > #16 0x7fcb69288c93 in PyEval_CallObjectWithKeywords (func=0x2527820, > > > > arg=0x7fcb6972f050, kw=) at Python/ceval.c:3663 > > > > #17 0x7fcb692ba7ba in t_bootstrap (boot_raw=0x250a820) at > > > > Modules/threadmodule.c:428 > > > > #18 0x7fcb68fa3851 in start_thread (arg=0x7fcb1bfff700) at > > > > pthread_create.c:301 > > > > #19 0x7fcb6866694d in clone () at > > > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 > > > > > > > > in pystack the threads were stuck in /usr/lib64/python2.6/threading.py > > > > (513): __bootstrap_inner > > > > > > > > in bootstrap_inner we use thread.allocate_lock which python-pthreading > > > > does > > > > not override. > > > > > > > > we suggest the following commit: > > > > > > > > From 9d89e9be1a379b3d93b23dd54a381b9ca0973ebc Mon Sep 17 00:00:00 2001 > > > > From: Yaniv Bronhaim > > > > Date: Mon, 3 Feb 2014 19:24:30 +0200 > > > > Subject: [PATCH] Mocking thread.allocate_lock with Lock imp > > > > > > > > Signed-off-by: Yaniv Bronhaim > > > > --- > > > > pthreading.py | 4 > > > > 1 file changed, 4 insertions(+) > > > > > > > > diff --git a/pthreading.py b/pthreading.py > > > > index 916ca7f..96df42c 100644 > > > > --- a/pthreading.py > > > > +++ b/pthreading.py > > > > @@ -132,6 +132,10 @@ def monkey_patch(): > > > > Thus, Queue and SocketServer can easily enjoy them. > > > > """ > > > > > > > > +import thread > > > > + > > > > +thread.allocate_lock = Lock > > > > + > > > > import threading > > > > > > > > threading.Condition = Condition > > > > -- > > > > 1.8.3.1 > > > > > > > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1022036 > > > > [2] https://bugzilla.redhat.com/show_bug.cgi?id=1060749 > > > > > > > > > > Replacing allocate_lock in thread is correct. However, since threading > > > copies > > > thread.allocate_lock, and you don't control import order, you have to > > > monkeypatch > > > threading.allocate_lock as well. > > > > > > The full should be: > > > > > > import thread > > > thread.allocate_lock = Lock > > > > > > import threading > > > threading.allocate_lock = Lock > > > threading.Lock = Lock > > > > > > Nir > > > > > > > thanks Nir for the reply and review > > I guess you meant threading._allocate_lock, but we monkey patch it in an > > order so we do control the import order.. not sure its necessary > > Correct, threading._allocate_lock. And we also have to patch > threading._active_limbo_lock. > > We don't control the user import order - If I import threading before > pthreading, both > threading._allocate_lock and threading._active_limbo_lock are using the > incorrect lock. > Since many library modules import threading, it is impossible to require any > import order. > > The safest way would be to patch all places that may use the original > thread.allocate_lock. The safest way would be to say loud and clear: the user must call pthreading.monkey_patch() on the begining of his main module, before threading is imported and most certainly before it is ever used. If we give the impression that doing anything else is fine, and we can end up replacing _active_limbo_lock while there's an entity (such as a _MainThread object) depending on its uniqueness. Dan. ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] suggested patch for python-pthreading
- Original Message - > From: "Dan Kenigsberg" > To: "Yaniv Bronheim" > Cc: "VDSM Project Development" , "Saggi > Mizrahi" > Sent: Tuesday, February 4, 2014 12:20:52 PM > Subject: Re: suggested patch for python-pthreading > > On Tue, Feb 04, 2014 at 04:04:37AM -0500, Yaniv Bronheim wrote: > > according to coredumps we found in the scope of the bug [1] we opened [2] > > that suggested to override python's implementation of thread.allocate_lock > > in each coredump we saw few threads stuck with the bt: > > > > #16 0x7fcb69288c93 in PyEval_CallObjectWithKeywords (func=0x2527820, > > arg=0x7fcb6972f050, kw=) at Python/ceval.c:3663 > > #17 0x7fcb692ba7ba in t_bootstrap (boot_raw=0x250a820) at > > Modules/threadmodule.c:428 > > #18 0x7fcb68fa3851 in start_thread (arg=0x7fcb1bfff700) at > > pthread_create.c:301 > > #19 0x7fcb6866694d in clone () at > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 > > > > in pystack the threads were stuck in /usr/lib64/python2.6/threading.py > > (513): __bootstrap_inner > > > > in bootstrap_inner we use thread.allocate_lock which python-pthreading does > > not override. > > > > we suggest the following commit: > > > > >From 9d89e9be1a379b3d93b23dd54a381b9ca0973ebc Mon Sep 17 00:00:00 2001 > > From: Yaniv Bronhaim > > Date: Mon, 3 Feb 2014 19:24:30 +0200 > > Subject: [PATCH] Mocking thread.allocate_lock with Lock imp > > > > Signed-off-by: Yaniv Bronhaim > > --- > > pthreading.py | 4 > > 1 file changed, 4 insertions(+) > > > > diff --git a/pthreading.py b/pthreading.py > > index 916ca7f..96df42c 100644 > > --- a/pthreading.py > > +++ b/pthreading.py > > @@ -132,6 +132,10 @@ def monkey_patch(): > > Thus, Queue and SocketServer can easily enjoy them. > > """ > > > > +import thread > > + > > +thread.allocate_lock = Lock > > + > > import threading > > > > threading.Condition = Condition > > -- > > 1.8.3.1 > > > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1022036 > > [2] https://bugzilla.redhat.com/show_bug.cgi?id=1060749 > > It makes sense to use pthreading.Lock for thread.allocate_lock instead of the > standard > threading.Lock CPU hog. However, I do not understand its > relevance to the deadlock sited above: pthreading.Lock fixes performance > issues, but not correctness issues, of threading.Lock. > > Would you explain, in the commit message of the pthreading patch, why > you believe that the implementation of thread.allocate_lock() is buggy? > Do you know if the bug is fixed in Python 3? > > Regards, > Dan. > We actually don't have concrete proof as we can't reproduce the bug so we can't test this. We are shooting in the dark hoping something hits. We assume it's there since all of our cordumps have a thread stuck acquiring the limbo lock. Since mixing lock implementations is probably a bad idea we assume that overriding this a thing we should do anyway we thought we'll give it a go. If VDSM gets stuck again we will have another coredump that we could compare to the others. ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] suggested patch for python-pthreading
On Tue, Feb 04, 2014 at 04:04:37AM -0500, Yaniv Bronheim wrote: > according to coredumps we found in the scope of the bug [1] we opened [2] > that suggested to override python's implementation of thread.allocate_lock > in each coredump we saw few threads stuck with the bt: > > #16 0x7fcb69288c93 in PyEval_CallObjectWithKeywords (func=0x2527820, > arg=0x7fcb6972f050, kw=) at Python/ceval.c:3663 > #17 0x7fcb692ba7ba in t_bootstrap (boot_raw=0x250a820) at > Modules/threadmodule.c:428 > #18 0x7fcb68fa3851 in start_thread (arg=0x7fcb1bfff700) at > pthread_create.c:301 > #19 0x7fcb6866694d in clone () at > ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 > > in pystack the threads were stuck in /usr/lib64/python2.6/threading.py > (513): __bootstrap_inner > > in bootstrap_inner we use thread.allocate_lock which python-pthreading does > not override. > > we suggest the following commit: > > >From 9d89e9be1a379b3d93b23dd54a381b9ca0973ebc Mon Sep 17 00:00:00 2001 > From: Yaniv Bronhaim > Date: Mon, 3 Feb 2014 19:24:30 +0200 > Subject: [PATCH] Mocking thread.allocate_lock with Lock imp > > Signed-off-by: Yaniv Bronhaim > --- > pthreading.py | 4 > 1 file changed, 4 insertions(+) > > diff --git a/pthreading.py b/pthreading.py > index 916ca7f..96df42c 100644 > --- a/pthreading.py > +++ b/pthreading.py > @@ -132,6 +132,10 @@ def monkey_patch(): > Thus, Queue and SocketServer can easily enjoy them. > """ > > +import thread > + > +thread.allocate_lock = Lock > + > import threading > > threading.Condition = Condition > -- > 1.8.3.1 > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1022036 > [2] https://bugzilla.redhat.com/show_bug.cgi?id=1060749 It makes sense to use pthreading.Lock for thread.allocate_lock instead of the standard threading.Lock CPU hog. However, I do not understand its relevance to the deadlock sited above: pthreading.Lock fixes performance issues, but not correctness issues, of threading.Lock. Would you explain, in the commit message of the pthreading patch, why you believe that the implementation of thread.allocate_lock() is buggy? Do you know if the bug is fixed in Python 3? Regards, Dan. ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] suggested patch for python-pthreading
- Original Message - > From: "Yaniv Bronheim" > To: "Nir Soffer" > Cc: "VDSM Project Development" > Sent: Tuesday, February 4, 2014 11:53:52 AM > Subject: Re: [vdsm] suggested patch for python-pthreading > > > > - Original Message - > > From: "Nir Soffer" > > To: "Yaniv Bronheim" > > Cc: "VDSM Project Development" > > Sent: Tuesday, February 4, 2014 11:39:56 AM > > Subject: Re: [vdsm] suggested patch for python-pthreading > > > > - Original Message - > > > From: "Yaniv Bronheim" > > > To: "VDSM Project Development" > > > Sent: Tuesday, February 4, 2014 11:04:37 AM > > > Subject: [vdsm] suggested patch for python-pthreading > > > > > > according to coredumps we found in the scope of the bug [1] we opened [2] > > > that suggested to override python's implementation of > > > thread.allocate_lock > > > in each coredump we saw few threads stuck with the bt: > > > > > > #16 0x7fcb69288c93 in PyEval_CallObjectWithKeywords (func=0x2527820, > > > arg=0x7fcb6972f050, kw=) at Python/ceval.c:3663 > > > #17 0x7fcb692ba7ba in t_bootstrap (boot_raw=0x250a820) at > > > Modules/threadmodule.c:428 > > > #18 0x7fcb68fa3851 in start_thread (arg=0x7fcb1bfff700) at > > > pthread_create.c:301 > > > #19 0x7fcb6866694d in clone () at > > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 > > > > > > in pystack the threads were stuck in /usr/lib64/python2.6/threading.py > > > (513): __bootstrap_inner > > > > > > in bootstrap_inner we use thread.allocate_lock which python-pthreading > > > does > > > not override. > > > > > > we suggest the following commit: > > > > > > From 9d89e9be1a379b3d93b23dd54a381b9ca0973ebc Mon Sep 17 00:00:00 2001 > > > From: Yaniv Bronhaim > > > Date: Mon, 3 Feb 2014 19:24:30 +0200 > > > Subject: [PATCH] Mocking thread.allocate_lock with Lock imp > > > > > > Signed-off-by: Yaniv Bronhaim > > > --- > > > pthreading.py | 4 > > > 1 file changed, 4 insertions(+) > > > > > > diff --git a/pthreading.py b/pthreading.py > > > index 916ca7f..96df42c 100644 > > > --- a/pthreading.py > > > +++ b/pthreading.py > > > @@ -132,6 +132,10 @@ def monkey_patch(): > > > Thus, Queue and SocketServer can easily enjoy them. > > > """ > > > > > > +import thread > > > + > > > +thread.allocate_lock = Lock > > > + > > > import threading > > > > > > threading.Condition = Condition > > > -- > > > 1.8.3.1 > > > > > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1022036 > > > [2] https://bugzilla.redhat.com/show_bug.cgi?id=1060749 > > > > > > > Replacing allocate_lock in thread is correct. However, since threading > > copies > > thread.allocate_lock, and you don't control import order, you have to > > monkeypatch > > threading.allocate_lock as well. > > > > The full should be: > > > > import thread > > thread.allocate_lock = Lock > > > > import threading > > threading.allocate_lock = Lock > > threading.Lock = Lock > > > > Nir > > > > thanks Nir for the reply and review > I guess you meant threading._allocate_lock, but we monkey patch it in an > order so we do control the import order.. not sure its necessary Correct, threading._allocate_lock. And we also have to patch threading._active_limbo_lock. We don't control the user import order - If I import threading before pthreading, both threading._allocate_lock and threading._active_limbo_lock are using the incorrect lock. Since many library modules import threading, it is impossible to require any import order. The safest way would be to patch all places that may use the original thread.allocate_lock. Nir ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] suggested patch for python-pthreading
- Original Message - > From: "Nir Soffer" > To: "Yaniv Bronheim" > Cc: "VDSM Project Development" > Sent: Tuesday, February 4, 2014 11:39:56 AM > Subject: Re: [vdsm] suggested patch for python-pthreading > > - Original Message - > > From: "Yaniv Bronheim" > > To: "VDSM Project Development" > > Sent: Tuesday, February 4, 2014 11:04:37 AM > > Subject: [vdsm] suggested patch for python-pthreading > > > > according to coredumps we found in the scope of the bug [1] we opened [2] > > that suggested to override python's implementation of thread.allocate_lock > > in each coredump we saw few threads stuck with the bt: > > > > #16 0x7fcb69288c93 in PyEval_CallObjectWithKeywords (func=0x2527820, > > arg=0x7fcb6972f050, kw=) at Python/ceval.c:3663 > > #17 0x7fcb692ba7ba in t_bootstrap (boot_raw=0x250a820) at > > Modules/threadmodule.c:428 > > #18 0x7fcb68fa3851 in start_thread (arg=0x7fcb1bfff700) at > > pthread_create.c:301 > > #19 0x7fcb6866694d in clone () at > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 > > > > in pystack the threads were stuck in /usr/lib64/python2.6/threading.py > > (513): __bootstrap_inner > > > > in bootstrap_inner we use thread.allocate_lock which python-pthreading does > > not override. > > > > we suggest the following commit: > > > > From 9d89e9be1a379b3d93b23dd54a381b9ca0973ebc Mon Sep 17 00:00:00 2001 > > From: Yaniv Bronhaim > > Date: Mon, 3 Feb 2014 19:24:30 +0200 > > Subject: [PATCH] Mocking thread.allocate_lock with Lock imp > > > > Signed-off-by: Yaniv Bronhaim > > --- > > pthreading.py | 4 > > 1 file changed, 4 insertions(+) > > > > diff --git a/pthreading.py b/pthreading.py > > index 916ca7f..96df42c 100644 > > --- a/pthreading.py > > +++ b/pthreading.py > > @@ -132,6 +132,10 @@ def monkey_patch(): > > Thus, Queue and SocketServer can easily enjoy them. > > """ > > > > +import thread > > + > > +thread.allocate_lock = Lock > > + > > import threading > > > > threading.Condition = Condition > > -- > > 1.8.3.1 > > > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1022036 > > [2] https://bugzilla.redhat.com/show_bug.cgi?id=1060749 > > > > Replacing allocate_lock in thread is correct. However, since threading copies > thread.allocate_lock, and you don't control import order, you have to > monkeypatch > threading.allocate_lock as well. > > The full should be: > > import thread > thread.allocate_lock = Lock > > import threading > threading.allocate_lock = Lock > threading.Lock = Lock > > Nir > thanks Nir for the reply and review I guess you meant threading._allocate_lock, but we monkey patch it in an order so we do control the import order.. not sure its necessary ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Re: [vdsm] suggested patch for python-pthreading
- Original Message - > From: "Yaniv Bronheim" > To: "VDSM Project Development" > Sent: Tuesday, February 4, 2014 11:04:37 AM > Subject: [vdsm] suggested patch for python-pthreading > > according to coredumps we found in the scope of the bug [1] we opened [2] > that suggested to override python's implementation of thread.allocate_lock > in each coredump we saw few threads stuck with the bt: > > #16 0x7fcb69288c93 in PyEval_CallObjectWithKeywords (func=0x2527820, > arg=0x7fcb6972f050, kw=) at Python/ceval.c:3663 > #17 0x7fcb692ba7ba in t_bootstrap (boot_raw=0x250a820) at > Modules/threadmodule.c:428 > #18 0x7fcb68fa3851 in start_thread (arg=0x7fcb1bfff700) at > pthread_create.c:301 > #19 0x7fcb6866694d in clone () at > ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 > > in pystack the threads were stuck in /usr/lib64/python2.6/threading.py > (513): __bootstrap_inner > > in bootstrap_inner we use thread.allocate_lock which python-pthreading does > not override. > > we suggest the following commit: > > From 9d89e9be1a379b3d93b23dd54a381b9ca0973ebc Mon Sep 17 00:00:00 2001 > From: Yaniv Bronhaim > Date: Mon, 3 Feb 2014 19:24:30 +0200 > Subject: [PATCH] Mocking thread.allocate_lock with Lock imp > > Signed-off-by: Yaniv Bronhaim > --- > pthreading.py | 4 > 1 file changed, 4 insertions(+) > > diff --git a/pthreading.py b/pthreading.py > index 916ca7f..96df42c 100644 > --- a/pthreading.py > +++ b/pthreading.py > @@ -132,6 +132,10 @@ def monkey_patch(): > Thus, Queue and SocketServer can easily enjoy them. > """ > > +import thread > + > +thread.allocate_lock = Lock > + > import threading > > threading.Condition = Condition > -- > 1.8.3.1 > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1022036 > [2] https://bugzilla.redhat.com/show_bug.cgi?id=1060749 > Replacing allocate_lock in thread is correct. However, since threading copies thread.allocate_lock, and you don't control import order, you have to monkeypatch threading.allocate_lock as well. The full should be: import thread thread.allocate_lock = Lock import threading threading.allocate_lock = Lock threading.Lock = Lock Nir ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
[vdsm] suggested patch for python-pthreading
according to coredumps we found in the scope of the bug [1] we opened [2] that suggested to override python's implementation of thread.allocate_lock in each coredump we saw few threads stuck with the bt: #16 0x7fcb69288c93 in PyEval_CallObjectWithKeywords (func=0x2527820, arg=0x7fcb6972f050, kw=) at Python/ceval.c:3663 #17 0x7fcb692ba7ba in t_bootstrap (boot_raw=0x250a820) at Modules/threadmodule.c:428 #18 0x7fcb68fa3851 in start_thread (arg=0x7fcb1bfff700) at pthread_create.c:301 #19 0x7fcb6866694d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 in pystack the threads were stuck in /usr/lib64/python2.6/threading.py (513): __bootstrap_inner in bootstrap_inner we use thread.allocate_lock which python-pthreading does not override. we suggest the following commit: From 9d89e9be1a379b3d93b23dd54a381b9ca0973ebc Mon Sep 17 00:00:00 2001 From: Yaniv Bronhaim Date: Mon, 3 Feb 2014 19:24:30 +0200 Subject: [PATCH] Mocking thread.allocate_lock with Lock imp Signed-off-by: Yaniv Bronhaim --- pthreading.py | 4 1 file changed, 4 insertions(+) diff --git a/pthreading.py b/pthreading.py index 916ca7f..96df42c 100644 --- a/pthreading.py +++ b/pthreading.py @@ -132,6 +132,10 @@ def monkey_patch(): Thus, Queue and SocketServer can easily enjoy them. """ +import thread + +thread.allocate_lock = Lock + import threading threading.Condition = Condition -- 1.8.3.1 [1] https://bugzilla.redhat.com/show_bug.cgi?id=1022036 [2] https://bugzilla.redhat.com/show_bug.cgi?id=1060749 Thanks, Yaniv Bronhaim. ___ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
