On Wed, Jun 1, 2016 at 1:44 PM, Nir Soffer <nsof...@redhat.com> wrote:
> Hi all, > > So we have this patch, aborting stuck tests and printing a backtrace > of all threads: > https://gerrit.ovirt.org/#/c/58212 > > On fedora, we get now a nice backtrace (see bellow) - which show that > we have unexpected > threads running during the tests. > > we have 25 iopproces threads like this: > > 09:09:20 Thread 34 (Thread 0x7fb5ab7fe700 (LWP 49858)): > 09:09:20 Traceback (most recent call first): > 09:09:20 File > "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in > NoIntrPoll > 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms > 09:09:20 File > "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in > _communicate > 09:09:20 pollres = NoIntrPoll(poller.poll, 5) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run > 09:09:20 self.__target(*self.__args, **self.__kwargs) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in > __bootstrap_inner > 09:09:20 self.run() > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in > __bootstrap > 09:09:20 self.__bootstrap_inner() > > This is expected given the current code managing these, and are daemon > threads. > > We have 8 reactor threads: > > 09:09:20 Thread 32 (Thread 0x7fb58effd700 (LWP 50091)): > 09:09:20 Traceback (most recent call first): > 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 192, in poll2 > 09:09:20 r = pollster.poll(timeout) > 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 220, in loop > 09:09:20 poll_fun(timeout, map) > 09:09:20 File > > "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py", > line 212, in process_requests > 09:09:20 count=1, > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run > 09:09:20 self.__target(*self.__args, **self.__kwargs) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in > __bootstrap_inner > 09:09:20 self.run() > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in > __bootstrap > 09:09:20 self.__bootstrap_inner() > > I don't know where these are started and it they are daemon threads. > > Searching the tests, we have too much code creating threads directly, > instead of using vdsm.concurrent.thread: > > $ git grep Thread tests/ | wc -l > 79 > > Please check your tests, and replace all threads to use > testlib.start_thread() > creating daemon threads. > > Code inheriting from threading.Thread is the worst, please keep a thread > instance instead, and start it with testlib.start_thread(). > > I sent this patch fixing sslTests for now: > https://gerrit.ovirt.org/58435 > > Nir > > ---- > > 09:06:05 testlibTests.TestStuckProcess > 09:09:19 test_stuck > 09:09:19 > ======================================================================== > 09:09:19 = Timeout completing tests - extracting stacktrace > = > 09:09:19 > ======================================================================== > 09:09:19 > 09:09:19 attach: No such file or directory. > 09:09:19 [New LWP 50476] > 09:09:19 [New LWP 50469] > 09:09:19 [New LWP 50462] > 09:09:19 [New LWP 50455] > 09:09:19 [New LWP 50448] > 09:09:19 [New LWP 50441] > 09:09:19 [New LWP 50434] > 09:09:19 [New LWP 50427] > 09:09:19 [New LWP 50420] > 09:09:19 [New LWP 50413] > 09:09:19 [New LWP 50406] > 09:09:19 [New LWP 50399] > 09:09:19 [New LWP 50392] > 09:09:19 [New LWP 50385] > 09:09:19 [New LWP 50378] > 09:09:19 [New LWP 50371] > 09:09:19 [New LWP 50364] > 09:09:19 [New LWP 50247] > 09:09:19 [New LWP 50240] > 09:09:19 [New LWP 50233] > 09:09:19 [New LWP 50226] > 09:09:19 [New LWP 50219] > 09:09:19 [New LWP 50216] > 09:09:19 [New LWP 50189] > 09:09:19 [New LWP 50175] > 09:09:19 [New LWP 50161] > 09:09:19 [New LWP 50147] > 09:09:19 [New LWP 50133] > 09:09:19 [New LWP 50119] > 09:09:19 [New LWP 50105] > 09:09:19 [New LWP 50091] > 09:09:19 [New LWP 49889] > 09:09:19 [New LWP 49858] > 09:09:19 [Thread debugging using libthread_db enabled] > 09:09:19 Using host libthread_db library "/lib64/libthread_db.so.1". > 09:09:20 0x00007fb5ffece8a3 in select () at > ../sysdeps/unix/syscall-template.S:84 > 09:09:20 84 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS) > 09:09:20 > 09:09:20 Thread 34 (Thread 0x7fb5ab7fe700 (LWP 49858)): > 09:09:20 Traceback (most recent call first): > 09:09:20 File > "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in > NoIntrPoll > 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms > 09:09:20 File > "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in > _communicate > 09:09:20 pollres = NoIntrPoll(poller.poll, 5) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run > 09:09:20 self.__target(*self.__args, **self.__kwargs) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in > __bootstrap_inner > 09:09:20 self.run() > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in > __bootstrap > 09:09:20 self.__bootstrap_inner() > 09:09:20 > 09:09:20 Thread 33 (Thread 0x7fb5a9899700 (LWP 49889)): > 09:09:20 Traceback (most recent call first): > 09:09:20 File > "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in > NoIntrPoll > 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms > 09:09:20 File > "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in > _communicate > 09:09:20 pollres = NoIntrPoll(poller.poll, 5) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run > 09:09:20 self.__target(*self.__args, **self.__kwargs) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in > __bootstrap_inner > 09:09:20 self.run() > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in > __bootstrap > 09:09:20 self.__bootstrap_inner() > 09:09:20 > 09:09:20 Thread 32 (Thread 0x7fb58effd700 (LWP 50091)): > 09:09:20 Traceback (most recent call first): > 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 192, in poll2 > 09:09:20 r = pollster.poll(timeout) > 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 220, in loop > 09:09:20 poll_fun(timeout, map) > 09:09:20 File > > "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py", > line 212, in process_requests > 09:09:20 count=1, > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run > 09:09:20 self.__target(*self.__args, **self.__kwargs) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in > __bootstrap_inner > 09:09:20 self.run() > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in > __bootstrap > 09:09:20 self.__bootstrap_inner() > 09:09:20 > 09:09:20 Thread 31 (Thread 0x7fb58e7fc700 (LWP 50105)): > 09:09:20 Traceback (most recent call first): > 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 192, in poll2 > 09:09:20 r = pollster.poll(timeout) > 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 220, in loop > 09:09:20 poll_fun(timeout, map) > 09:09:20 File > > "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py", > line 212, in process_requests > 09:09:20 count=1, > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run > 09:09:20 self.__target(*self.__args, **self.__kwargs) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in > __bootstrap_inner > 09:09:20 self.run() > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in > __bootstrap > 09:09:20 self.__bootstrap_inner() > 09:09:20 > 09:09:20 Thread 30 (Thread 0x7fb58dffb700 (LWP 50119)): > 09:09:20 Traceback (most recent call first): > 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 192, in poll2 > 09:09:20 r = pollster.poll(timeout) > 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 220, in loop > 09:09:20 poll_fun(timeout, map) > 09:09:20 File > > "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py", > line 212, in process_requests > 09:09:20 count=1, > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run > 09:09:20 self.__target(*self.__args, **self.__kwargs) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in > __bootstrap_inner > 09:09:20 self.run() > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in > __bootstrap > 09:09:20 self.__bootstrap_inner() > 09:09:20 > 09:09:20 Thread 29 (Thread 0x7fb58d7fa700 (LWP 50133)): > 09:09:20 Traceback (most recent call first): > 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 192, in poll2 > 09:09:20 r = pollster.poll(timeout) > 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 220, in loop > 09:09:20 poll_fun(timeout, map) > 09:09:20 File > > "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py", > line 212, in process_requests > 09:09:20 count=1, > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run > 09:09:20 self.__target(*self.__args, **self.__kwargs) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in > __bootstrap_inner > 09:09:20 self.run() > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in > __bootstrap > 09:09:20 self.__bootstrap_inner() > 09:09:20 > 09:09:20 Thread 28 (Thread 0x7fb58cff9700 (LWP 50147)): > 09:09:20 Traceback (most recent call first): > 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 192, in poll2 > 09:09:20 r = pollster.poll(timeout) > 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 220, in loop > 09:09:20 poll_fun(timeout, map) > 09:09:20 File > > "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py", > line 212, in process_requests > 09:09:20 count=1, > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run > 09:09:20 self.__target(*self.__args, **self.__kwargs) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in > __bootstrap_inner > 09:09:20 self.run() > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in > __bootstrap > 09:09:20 self.__bootstrap_inner() > 09:09:20 > 09:09:20 Thread 27 (Thread 0x7fb56bfff700 (LWP 50161)): > 09:09:20 Traceback (most recent call first): > 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 192, in poll2 > 09:09:20 r = pollster.poll(timeout) > 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 220, in loop > 09:09:20 poll_fun(timeout, map) > 09:09:20 File > > "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py", > line 212, in process_requests > 09:09:20 count=1, > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run > 09:09:20 self.__target(*self.__args, **self.__kwargs) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in > __bootstrap_inner > 09:09:20 self.run() > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in > __bootstrap > 09:09:20 self.__bootstrap_inner() > 09:09:20 > 09:09:20 Thread 26 (Thread 0x7fb56b7fe700 (LWP 50175)): > 09:09:20 Traceback (most recent call first): > 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 192, in poll2 > 09:09:20 r = pollster.poll(timeout) > 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 220, in loop > 09:09:20 poll_fun(timeout, map) > 09:09:20 File > > "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py", > line 212, in process_requests > 09:09:20 count=1, > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run > 09:09:20 self.__target(*self.__args, **self.__kwargs) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in > __bootstrap_inner > 09:09:20 self.run() > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in > __bootstrap > 09:09:20 self.__bootstrap_inner() > 09:09:20 > 09:09:20 Thread 25 (Thread 0x7fb56affd700 (LWP 50189)): > 09:09:20 Traceback (most recent call first): > 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 192, in poll2 > 09:09:20 r = pollster.poll(timeout) > 09:09:20 File "/usr/lib64/python2.7/asyncore.py", line 220, in loop > 09:09:20 poll_fun(timeout, map) > 09:09:20 File > > "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/lib/yajsonrpc/betterAsyncore.py", > line 212, in process_requests > 09:09:20 count=1, > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run > 09:09:20 self.__target(*self.__args, **self.__kwargs) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in > __bootstrap_inner > 09:09:20 self.run() > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in > __bootstrap > 09:09:20 self.__bootstrap_inner() > 09:09:20 > 09:09:20 Thread 24 (Thread 0x7fb59affd700 (LWP 50216)): > 09:09:20 Traceback (most recent call first): > 09:09:20 File > "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in > NoIntrPoll > 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms > 09:09:20 File > "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in > _communicate > 09:09:20 pollres = NoIntrPoll(poller.poll, 5) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run > 09:09:20 self.__target(*self.__args, **self.__kwargs) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in > __bootstrap_inner > 09:09:20 self.run() > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in > __bootstrap > 09:09:20 self.__bootstrap_inner() > 09:09:20 > 09:09:20 Thread 23 (Thread 0x7fb5e905f700 (LWP 50219)): > 09:09:20 Traceback (most recent call first): > 09:09:20 File > "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in > NoIntrPoll > 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms > 09:09:20 File > "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in > _communicate > 09:09:20 pollres = NoIntrPoll(poller.poll, 5) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run > 09:09:20 self.__target(*self.__args, **self.__kwargs) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in > __bootstrap_inner > 09:09:20 self.run() > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in > __bootstrap > 09:09:20 self.__bootstrap_inner() > 09:09:20 > 09:09:20 Thread 22 (Thread 0x7fb59bfff700 (LWP 50226)): > 09:09:20 Traceback (most recent call first): > 09:09:20 File > "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in > NoIntrPoll > 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms > 09:09:20 File > "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in > _communicate > 09:09:20 pollres = NoIntrPoll(poller.poll, 5) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run > 09:09:20 self.__target(*self.__args, **self.__kwargs) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in > __bootstrap_inner > 09:09:20 self.run() > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in > __bootstrap > 09:09:20 self.__bootstrap_inner() > 09:09:20 > 09:09:20 Thread 21 (Thread 0x7fb5e885e700 (LWP 50233)): > 09:09:20 Traceback (most recent call first): > 09:09:20 File > "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in > NoIntrPoll > 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms > 09:09:20 File > "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in > _communicate > 09:09:20 pollres = NoIntrPoll(poller.poll, 5) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run > 09:09:20 self.__target(*self.__args, **self.__kwargs) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in > __bootstrap_inner > 09:09:20 self.run() > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in > __bootstrap > 09:09:20 self.__bootstrap_inner() > 09:09:20 > 09:09:20 Thread 20 (Thread 0x7fb5e9860700 (LWP 50240)): > 09:09:20 Traceback (most recent call first): > 09:09:20 File > "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in > NoIntrPoll > 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms > 09:09:20 File > "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in > _communicate > 09:09:20 pollres = NoIntrPoll(poller.poll, 5) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run > 09:09:20 self.__target(*self.__args, **self.__kwargs) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in > __bootstrap_inner > 09:09:20 self.run() > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in > __bootstrap > 09:09:20 self.__bootstrap_inner() > 09:09:20 > 09:09:20 Thread 19 (Thread 0x7fb5eb864700 (LWP 50247)): > 09:09:20 Traceback (most recent call first): > 09:09:20 File > "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in > NoIntrPoll > 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms > 09:09:20 File > "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in > _communicate > 09:09:20 pollres = NoIntrPoll(poller.poll, 5) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run > 09:09:20 self.__target(*self.__args, **self.__kwargs) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in > __bootstrap_inner > 09:09:20 self.run() > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in > __bootstrap > 09:09:20 self.__bootstrap_inner() > 09:09:20 > 09:09:20 Thread 18 (Thread 0x7fb5c9061700 (LWP 50364)): > 09:09:20 Traceback (most recent call first): > 09:09:20 File > "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in > NoIntrPoll > 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms > 09:09:20 File > "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in > _communicate > 09:09:20 pollres = NoIntrPoll(poller.poll, 5) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run > 09:09:20 self.__target(*self.__args, **self.__kwargs) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in > __bootstrap_inner > 09:09:20 self.run() > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in > __bootstrap > 09:09:20 self.__bootstrap_inner() > 09:09:20 > 09:09:20 Thread 17 (Thread 0x7fb5cb7fe700 (LWP 50371)): > 09:09:20 Traceback (most recent call first): > 09:09:20 File > "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in > NoIntrPoll > 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms > 09:09:20 File > "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in > _communicate > 09:09:20 pollres = NoIntrPoll(poller.poll, 5) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run > 09:09:20 self.__target(*self.__args, **self.__kwargs) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in > __bootstrap_inner > 09:09:20 self.run() > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in > __bootstrap > 09:09:20 self.__bootstrap_inner() > 09:09:20 > 09:09:20 Thread 16 (Thread 0x7fb5cbfff700 (LWP 50378)): > 09:09:20 Traceback (most recent call first): > 09:09:20 File > "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in > NoIntrPoll > 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms > 09:09:20 File > "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in > _communicate > 09:09:20 pollres = NoIntrPoll(poller.poll, 5) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run > 09:09:20 self.__target(*self.__args, **self.__kwargs) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in > __bootstrap_inner > 09:09:20 self.run() > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in > __bootstrap > 09:09:20 self.__bootstrap_inner() > 09:09:20 > 09:09:20 Thread 15 (Thread 0x7fb5ea862700 (LWP 50385)): > 09:09:20 Traceback (most recent call first): > 09:09:20 File > "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in > NoIntrPoll > 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms > 09:09:20 File > "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in > _communicate > 09:09:20 pollres = NoIntrPoll(poller.poll, 5) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run > 09:09:20 self.__target(*self.__args, **self.__kwargs) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in > __bootstrap_inner > 09:09:20 self.run() > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in > __bootstrap > 09:09:20 self.__bootstrap_inner() > 09:09:20 > 09:09:20 Thread 14 (Thread 0x7fb5eb063700 (LWP 50392)): > 09:09:20 Traceback (most recent call first): > 09:09:20 File > "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in > NoIntrPoll > 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms > 09:09:20 File > "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in > _communicate > 09:09:20 pollres = NoIntrPoll(poller.poll, 5) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run > 09:09:20 self.__target(*self.__args, **self.__kwargs) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in > __bootstrap_inner > 09:09:20 self.run() > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in > __bootstrap > 09:09:20 self.__bootstrap_inner() > 09:09:20 > 09:09:20 Thread 13 (Thread 0x7fb5aaffd700 (LWP 50399)): > 09:09:20 Traceback (most recent call first): > 09:09:20 File > "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in > NoIntrPoll > 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms > 09:09:20 File > "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in > _communicate > 09:09:20 pollres = NoIntrPoll(poller.poll, 5) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run > 09:09:20 self.__target(*self.__args, **self.__kwargs) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in > __bootstrap_inner > 09:09:20 self.run() > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in > __bootstrap > 09:09:20 self.__bootstrap_inner() > 09:09:20 > 09:09:20 Thread 12 (Thread 0x7fb5a8f18700 (LWP 50406)): > 09:09:20 Traceback (most recent call first): > 09:09:20 File > "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in > NoIntrPoll > 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms > 09:09:20 File > "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in > _communicate > 09:09:20 pollres = NoIntrPoll(poller.poll, 5) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run > 09:09:20 self.__target(*self.__args, **self.__kwargs) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in > __bootstrap_inner > 09:09:20 self.run() > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in > __bootstrap > 09:09:20 self.__bootstrap_inner() > 09:09:20 > 09:09:20 Thread 11 (Thread 0x7fb59b7fe700 (LWP 50413)): > 09:09:20 Traceback (most recent call first): > 09:09:20 File > "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in > NoIntrPoll > 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms > 09:09:20 File > "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in > _communicate > 09:09:20 pollres = NoIntrPoll(poller.poll, 5) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run > 09:09:20 self.__target(*self.__args, **self.__kwargs) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in > __bootstrap_inner > 09:09:20 self.run() > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in > __bootstrap > 09:09:20 self.__bootstrap_inner() > 09:09:20 > 09:09:20 Thread 10 (Thread 0x7fb59a7fc700 (LWP 50420)): > 09:09:20 Traceback (most recent call first): > 09:09:20 File > "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in > NoIntrPoll > 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms > 09:09:20 File > "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in > _communicate > 09:09:20 pollres = NoIntrPoll(poller.poll, 5) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run > 09:09:20 self.__target(*self.__args, **self.__kwargs) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in > __bootstrap_inner > 09:09:20 self.run() > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in > __bootstrap > 09:09:20 self.__bootstrap_inner() > 09:09:20 > 09:09:20 Thread 9 (Thread 0x7fb599ffb700 (LWP 50427)): > 09:09:20 Traceback (most recent call first): > 09:09:20 File > "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in > NoIntrPoll > 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms > 09:09:20 File > "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in > _communicate > 09:09:20 pollres = NoIntrPoll(poller.poll, 5) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run > 09:09:20 self.__target(*self.__args, **self.__kwargs) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in > __bootstrap_inner > 09:09:20 self.run() > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in > __bootstrap > 09:09:20 self.__bootstrap_inner() > 09:09:20 > 09:09:20 Thread 8 (Thread 0x7fb5997fa700 (LWP 50434)): > 09:09:20 Traceback (most recent call first): > 09:09:20 File > "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in > NoIntrPoll > 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms > 09:09:20 File > "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in > _communicate > 09:09:20 pollres = NoIntrPoll(poller.poll, 5) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run > 09:09:20 self.__target(*self.__args, **self.__kwargs) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in > __bootstrap_inner > 09:09:20 self.run() > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in > __bootstrap > 09:09:20 self.__bootstrap_inner() > 09:09:20 > 09:09:20 Thread 7 (Thread 0x7fb598ff9700 (LWP 50441)): > 09:09:20 Traceback (most recent call first): > 09:09:20 File > "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in > NoIntrPoll > 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms > 09:09:20 File > "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in > _communicate > 09:09:20 pollres = NoIntrPoll(poller.poll, 5) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run > 09:09:20 self.__target(*self.__args, **self.__kwargs) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in > __bootstrap_inner > 09:09:20 self.run() > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in > __bootstrap > 09:09:20 self.__bootstrap_inner() > 09:09:20 > 09:09:20 Thread 6 (Thread 0x7fb58ffff700 (LWP 50448)): > 09:09:20 Traceback (most recent call first): > 09:09:20 File > "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in > NoIntrPoll > 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms > 09:09:20 File > "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in > _communicate > 09:09:20 pollres = NoIntrPoll(poller.poll, 5) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run > 09:09:20 self.__target(*self.__args, **self.__kwargs) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in > __bootstrap_inner > 09:09:20 self.run() > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in > __bootstrap > 09:09:20 self.__bootstrap_inner() > 09:09:20 > 09:09:20 Thread 5 (Thread 0x7fb58f7fe700 (LWP 50455)): > 09:09:20 Traceback (most recent call first): > 09:09:20 File > "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in > NoIntrPoll > 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms > 09:09:20 File > "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in > _communicate > 09:09:20 pollres = NoIntrPoll(poller.poll, 5) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run > 09:09:20 self.__target(*self.__args, **self.__kwargs) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in > __bootstrap_inner > 09:09:20 self.run() > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in > __bootstrap > 09:09:20 self.__bootstrap_inner() > 09:09:20 > 09:09:20 Thread 4 (Thread 0x7fb56a7fc700 (LWP 50462)): > 09:09:20 Traceback (most recent call first): > 09:09:20 File > "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in > NoIntrPoll > 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms > 09:09:20 File > "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in > _communicate > 09:09:20 pollres = NoIntrPoll(poller.poll, 5) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run > 09:09:20 self.__target(*self.__args, **self.__kwargs) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in > __bootstrap_inner > 09:09:20 self.run() > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in > __bootstrap > 09:09:20 self.__bootstrap_inner() > 09:09:20 > 09:09:20 Thread 3 (Thread 0x7fb569ffb700 (LWP 50469)): > 09:09:20 Traceback (most recent call first): > 09:09:20 File > "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in > NoIntrPoll > 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms > 09:09:20 File > "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in > _communicate > 09:09:20 pollres = NoIntrPoll(poller.poll, 5) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run > 09:09:20 self.__target(*self.__args, **self.__kwargs) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in > __bootstrap_inner > 09:09:20 self.run() > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in > __bootstrap > 09:09:20 self.__bootstrap_inner() > 09:09:20 > 09:09:20 Thread 2 (Thread 0x7fb5697fa700 (LWP 50476)): > 09:09:20 Traceback (most recent call first): > 09:09:20 File > "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 222, in > NoIntrPoll > 09:09:20 return pollfun(timeout * 1000) # timeout for poll is in ms > 09:09:20 File > "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 115, in > _communicate > 09:09:20 pollres = NoIntrPoll(poller.poll, 5) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 757, in run > 09:09:20 self.__target(*self.__args, **self.__kwargs) > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 804, in > __bootstrap_inner > 09:09:20 self.run() > 09:09:20 File "/usr/lib64/python2.7/threading.py", line 777, in > __bootstrap > 09:09:20 self.__bootstrap_inner() > 09:09:20 > 09:09:20 Thread 1 (Thread 0x7fb60109d700 (LWP 49417)): > 09:09:20 Traceback (most recent call first): > 09:09:20 File > > "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/testlibTests.py", > line 264, in test_stuck > 09:09:20 time.sleep(300) > 09:09:20 File "/usr/lib64/python2.7/unittest/case.py", line 367, in run > 09:09:20 testMethod() > 09:09:20 File "/usr/lib64/python2.7/unittest/case.py", line 431, in > __call__ > 09:09:20 return self.run(*args, **kwds) > 09:09:20 File "/usr/lib/python2.7/site-packages/nose/case.py", line > 151, in runTest > 09:09:20 test(result) > 09:09:20 File "/usr/lib/python2.7/site-packages/nose/case.py", line > 133, in run > 09:09:20 self.runTest(result) > 09:09:20 File "/usr/lib/python2.7/site-packages/nose/case.py", line > 45, in __call__ > 09:09:20 return self.run(*arg, **kwarg) > 09:09:20 File "/usr/lib/python2.7/site-packages/nose/suite.py", line > 224, in run > 09:09:20 test(orig) > 09:09:20 File "/usr/lib/python2.7/site-packages/nose/suite.py", line > 177, in __call__ > 09:09:20 return self.run(*arg, **kw) > 09:09:20 File "/usr/lib/python2.7/site-packages/nose/suite.py", line > 224, in run > 09:09:20 test(orig) > 09:09:20 File "/usr/lib/python2.7/site-packages/nose/suite.py", line > 177, in __call__ > 09:09:20 return self.run(*arg, **kw) > 09:09:20 File "/usr/lib/python2.7/site-packages/nose/suite.py", line > 224, in run > 09:09:20 test(orig) > 09:09:20 File "/usr/lib/python2.7/site-packages/nose/suite.py", line > 177, in __call__ > 09:09:20 return self.run(*arg, **kw) > 09:09:20 File "/usr/lib/python2.7/site-packages/nose/suite.py", line > 224, in run > 09:09:20 test(orig) > 09:09:20 File "/usr/lib/python2.7/site-packages/nose/suite.py", line > 177, in __call__ > 09:09:20 return self.run(*arg, **kw) > 09:09:20 File "/usr/lib/python2.7/site-packages/nose/core.py", line 62, > in run > 09:09:20 test(result) > 09:09:20 File "/usr/lib/python2.7/site-packages/nose/core.py", line > 207, in runTests > 09:09:20 result = self.testRunner.run(self.test) > 09:09:20 File "/usr/lib64/python2.7/unittest/main.py", line 95, in > __init__ > 09:09:20 self.runTests() > 09:09:20 File "/usr/lib/python2.7/site-packages/nose/core.py", line > 121, in __init__ > 09:09:20 **extra_args) > 09:09:20 File "/usr/lib/python2.7/site-packages/nose/core.py", line > 301, in run > 09:09:20 return TestProgram(*arg, **kw).success > 09:09:20 File > > "/home/jenkins/workspace/vdsm_master_check-patch-fc23-x86_64/vdsm/tests/testlib.py", > line 396, in run > 09:09:20 sys.exit(not core.run(config=conf, testRunner=runner, > argv=argv)) > 09:09:20 File "../tests/testrunner.py", line 88, in <module> > 09:09:20 testlib.run() > 09:09:20 > 09:09:20 > ======================================================================== > 09:09:20 = Aborting tests > = > 09:09:20 > ======================================================================== > _______________________________________________ > Devel mailing list > Devel@ovirt.org > http://lists.ovirt.org/mailman/listinfo/devel > Thank you for pointing this out, in networking there is one direct usage.. taking it out. Thanks, Edy.
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel