[issue43802] Seg fault on macOS using multiprocessing.JoinableQueue

[Jacob Walls]

Jacob Walls  added the comment:

Unfortunately, at the outset I should have tested this without multiprocessing. 
I can reproduce without multiprocessing[1], which meant I could more easily 
pinpoint the failure. There is an expensive O(nm) algorithm[2] in the music21 
library that is overflowing.

I appreciate your time looking into this. Closing.

Regards, Jacob


[1] in the provided script, after one call to lc.save() call 
lc.rebuildMetadataCache(useMultiprocessing=False)

[2] music21.analysis.discrete.Ambitus.getPitchRanges(), and I plan to do 
something about it.

--
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43802] Seg fault on macOS using multiprocessing.JoinableQueue

[Jacob Walls]

Jacob Walls  added the comment:

Thanks for this detailed reply. I reproduced on Python 3.9.4 on the same iMac 
from my original report running macOS 10.13.6, but with much lesser frequency 
(I wouldn't use the word "consistently" anymore).

I tried on a MacBook Pro with worn-out hardware running a newer OS (10.15.4) 
and could not reproduce the issue there. I also built cPython (Python 3.10.0a7+ 
(heads/master:ac05f82ad4, Apr 10 2021, 20:16:36) [Clang 10.0.0 
(clang-1000.10.44.4)] on darwin) using --with-pydebug and ran the test case a 
few times on the good-hardware iMac, and observed the file parsing 
(predictably) slow to a crawl, but no reproduction of the segfault. This leads 
me to believe that, yes, this is a race condition I'm encountering on fast 
hardware.

Possibly related to issue-25769, since music21 makes heavy use of weakrefs and 
since music21.metadata.caching.MetadataCachingJob.run() calls gc.collect(). 
Perhaps I can look into engineering a minimal test case based on that 
discussion, involving a deliberately expensive __eq__() call.

To answer your original question: my first report on 3.9.2 was on this specific 
version:
3.9.2 (v3.9.2:1a79785e3e, Feb 19 2021, 09:06:10) \n[Clang 6.0 (clang-600.0.57)]

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43802] Seg fault on macOS using multiprocessing.JoinableQueue

[Ned Deily]

Ned Deily  added the comment:

Thanks for providing a detailed and relatively simple-to-run test case for such 
a complicated failure. Not totally surprising for what appears to likely be a 
race condition, I have been unable to reproduce it under several macOS 
environments including in a 10.13.6 VM with multiple cores and under 11.2.3. 
I'm not sure if this would be expected to affect the results but I did receive 
multiple "WARNING: Could not import wedge: Error in getting DynamicWedges" 
messages when running the test case.

Doing a quick exam of the installed set up, it appears that there is no attempt 
to use multiprocessing's "fork" method which is known to be problematic on 
macOS so that's a plus. And there don't appear to be any extension modules so 
the test case is pure Python, eliminating other likely suspects.

One question that does come to mind is exactly which version of Python 3.9.2 
you are testing with; can you provide the results of: /path/to/python3.9 -c 
'import sys;print(sys.version)' ?

Searching bugs.python.org, I see a few open issues with segfaults in 
PyObject_RichCompare but nothing that leaps out as being obviously similar. If 
it were possible to reproduce the segfault in other environments, like with 
3.9.4 or on newer versions of macOS or on a current Linux platform, that would 
help to confirm the issue. Even better would be to be able to reproduce the 
issue while running a current Python 3.9 built with --with-pydebug on; 
unfortunately, we don't normally provide pre-built debug binaries on 
python.org. And, of course, running in debug mode could affect the rece 
condition, if that is indeed the issue.

--
nosy: +davin, pitrou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43802] Seg fault on macOS using multiprocessing.JoinableQueue

[Jacob Walls]

New submission from Jacob Walls :

macOS 10.13.6
Python 3.9.2

I can consistently reproduce a seg fault while using 
multiprocessing.JoinableQueue in Python 3.9.2.

My use case is the sheet music processing library music21. My fork includes a 
folder of 209 files I use to reproduce, running 3 cores, shown in the script 
below. (This is a subset of the over 1,000 files found here: 
https://github.com/MarkGotham/When-in-Rome/tree/master/Corpus/OpenScore-LiederCorpus
Using this set of 1,000 files reproduces nearly every time; using the 209 files 
I committed to my fork was enough to reproduce about 75% of the time.)

I'm a contributor to music21, so if this is an overwhelming amount of 
information to debug, I will gladly pare this down as much as I can or create 
some methods to access the multiprocessing functionality more directly. Many 
thanks for any assistance.


pip3 install 
git+https://github.com/jacobtylerwalls/music21.git@bpo-investigation

from music21 import corpus
# suggest using a unique name each attempt
lc = corpus.corpora.LocalCorpus(name='bpo-investigation')
# point to the directory of files I committed to my fork for this investigation
lc.addPath('/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/music21/bpo-files')
# parse the files using multiprocessing
# calls music21.metadata.bundles.MetadataBundle.addFromPaths()
# which calls music21.metadata.caching.process_parallel()
lc.save()

# CTRL-C to recover from seg fault
# then, wipe out the entries in .music21rc so that you can cleanly reproduce 
again
from music21 import environment
us = environment.UserSettings()
us['localCorporaSettings'] = {}
quit()


Process:   Python [31677]
Path:  
/Library/Frameworks/Python.framework/Versions/3.9/Resources/Python.app/Contents/MacOS/Python
Identifier:Python
Version:   3.9.2 (3.9.2)
Code Type: X86-64 (Native)
Parent Process:Python [31674]
Responsible:   Python [31677]
User ID:   501

Date/Time: 2021-04-10 11:21:19.294 -0400
OS Version:Mac OS X 10.13.6 (17G14042)
Report Version:12
Anonymous UUID:E7B0208A-19D6-ABDF-B3EA-3910A56B3E72

Sleep/Wake UUID:   C4B83F57-6AD1-469E-82AE-88214FAA6283

Time Awake Since Boot: 14 seconds
Time Since Wake:   5900 seconds

System Integrity Protection: enabled

Crashed Thread:0  Dispatch queue: com.apple.main-thread

Exception Type:EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:   KERN_INVALID_ADDRESS at 0x000100b3acd8
Exception Note:EXC_CORPSE_NOTIFY

Termination Signal:Segmentation fault: 11
Termination Reason:Namespace SIGNAL, Code 0xb
Terminating Process:   exc handler [0]

VM Regions Near 0x100b3acd8:
--> 
__TEXT 0001068bb000-0001068bc000 [4K] r-x/rwx 
SM=COW   
[/Library/Frameworks/Python.framework/Versions/3.9/Resources/Python.app/Contents/MacOS/Python]

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   org.python.python   0x000106944072 PyObject_RichCompare 
+ 258
1   org.python.python   0x000106943e9b 
PyObject_RichCompareBool + 43
2   org.python.python   0x0001069ce3c0 min_max + 624
3   org.python.python   0x000106940bab cfunction_call + 59
4   org.python.python   0x000106901cad _PyObject_MakeTpCall 
+ 365
5   org.python.python   0x0001069d865c call_function + 876
6   org.python.python   0x0001069d5b8b 
_PyEval_EvalFrameDefault + 25371
7   org.python.python   0x000106902478 
function_code_fastcall + 104
8   org.python.python   0x0001069d85cc call_function + 732
9   org.python.python   0x0001069d5ad2 
_PyEval_EvalFrameDefault + 25186
10  org.python.python   0x0001069d92c3 _PyEval_EvalCode + 
2611
11  org.python.python   0x000106902401 
_PyFunction_Vectorcall + 289
12  org.python.python   0x0001069d85cc call_function + 732
13  org.python.python   0x0001069d5ad2 
_PyEval_EvalFrameDefault + 25186
14  org.python.python   0x0001069d92c3 _PyEval_EvalCode + 
2611
15  org.python.python   0x000106902401 
_PyFunction_Vectorcall + 289
16  org.python.python   0x000106901b05 
_PyObject_FastCallDictTstate + 293
17  org.python.python   0x0001069026e8 
_PyObject_Call_Prepend + 152
18  org.python.python   0x00010695be85 slot_tp_init + 165
19  org.python.python   0x0001069573d9 type_call + 345
20  org.python.python   0x000106901cad _PyObject_MakeTpCall 
+ 365
21  org.python.python   0x0001069d865c call_function + 876
22  org.python.python   0x0001069d5af3 
_PyEval_EvalFrameDefault