I’m hitting an issue with deadlocks on our fork of the edk2 build system. It
only happens on our builders (30+ cores), and I can’t repo locally. We realized
we had fixed the number of threads, and when we removed that from our build
config we started seeing this issue.
The build is deadlocking during the `…` so that implies during the autogen
phase. I hooked up a signal handler so I could catch `kill != 9` and dump the
Python threads.
I see these threads:
MainThread: seems stuck at the self.AutoGenMgr.join() in StartAutoGen().
Thread-1: looks like a logging thread.
Thread-2: looks like the thread printing …
2 x QueueFeederThread: I’m not sure if these are workers for Python or kicked
off by the build?
I’m looking for some advice on how to debug the deadlock. I’ve tried sampling
via kill -2 etc. and my exception handler but I don’t seem to be able to see
the work happening on Python threads….
Thread: QueueFeederThread(123145563672576)
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py:774
function __bootstrap
self.__bootstrap_inner()
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py:801
function __bootstrap_inner
self.run()
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py:754
function run
self.__target(*self.__args, **self.__kwargs)
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py:252
function _feed
nwait()
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py:340
function wait
waiter.acquire()
Thread: Thread-2(123145555259392)
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py:774
function __bootstrap
self.__bootstrap_inner()
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py:801
function __bootstrap_inner
self.run()
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py:754
function run
self.__target(*self.__args, **self.__kwargs)
./BaseTools/Source/Python/Common/Misc.py:907 function _ProgressThreadEntry
time.sleep(self._CheckInterval)
Thread: MainThread(4614635008)
./BaseTools/Source/Python/build/build.py:2754 function <module>
r = Main()
./BaseTools/Source/Python/build/build.py:2643 function Main
MyBuild.Launch()
./BaseTools/Source/Python/build/build.py:2438 function Launch
self._MultiThreadBuildPlatform()
./BaseTools/Source/Python/build/build.py:2246 function
_MultiThreadBuildPlatform
Wa, self.BuildModules = self.PerformAutoGen(BuildTarget,ToolChain)
./BaseTools/Source/Python/build/build.py:2185 function PerformAutoGen
autogen_rt, errorcode = self.StartAutoGen(mqueue, Pa.DataPipe,
self.SkipAutoGen, PcdMaList, cqueue)
./BaseTools/Source/Python/build/build.py:889 function StartAutoGen
self.AutoGenMgr.join()
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py:940
function join
self.__block.wait()
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py:340
function wait
waiter.acquire()
./BaseTools/Source/Python/build/build.py:2737 function AppleDumpAllThreadStacks
for filename, lineno, name, line in traceback.extract_stack(stack):
Thread: Thread-1(123145546846208)
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py:774
function __bootstrap
self.__bootstrap_inner()
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py:801
function __bootstrap_inner
self.run()
./BaseTools/Source/Python/AutoGen/AutoGenWorker.py:85 function run
log_message = self.log_q.get()
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py:117
function get
res = self._recv()
Thread: QueueFeederThread(123145551052800)
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py:774
function __bootstrap
self.__bootstrap_inner()
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py:801
function __bootstrap_inner
self.run()
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py:754
function run
self.__target(*self.__args, **self.__kwargs)
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py:252
function _feed
nwait()
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py:340
function wait
waiter.acquire()
Thanks,
Andrew Fish
-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#77094): https://edk2.groups.io/g/devel/message/77094
Mute This Topic: https://groups.io/mt/83793143/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
