I updated the bit about OpenBLAS wheel with some more information on the status of that work. It's not super important, but FYI.
I also want to disagree with this characterization of the Accelerate/multiprocessing issue: "This problem was due to a bug in multiprocessing and is fixed in Python 3.4 and later; Accelerate was POSIX compliant but multiprocessing was not." In 3.4 it became possible to *work around* this issue, but it requires configuring the multiprocessing module in a non-default way, which means that the common end-user experience is still that they try using multiprocessing, and they get random hangs with no other feedback, and then spend hours or days debugging before they discover this configuration option. (And the problem occurs on MacOS only, so you get extra fun when e.g. a module is developed on Windows or Linux and then you give it to a less-technical collaborator on MacOS and it breaks on their computer and you have no idea why.) And the workaround is suboptimal -- fork()'s memory-sharing semantics are very powerful. I've had cases where I could easily and efficiently solve a problem using multiprocessing in fork() mode, but where enabling the workaround for Accelerate would have made it impossible. (Specifically this happened because I had a huge read-only data structure that I could load once in the parent process, and then all the child processes could share it through fork's virtual memory magic; I didn't have enough memory to load two copies of it, yet fork let me have 10 or 20 virtual copies.) Technically, yes, mixing threads and fork can't be done in a POSIX-compliant manner. But no-one runs their code on an abstract POSIX machine, and on actual systems it's totally possible to make this work reliably. OpenBLAS does it. Users don't care if Apple is technically correct, they just want their stuff to work. -n On Tue, Jul 25, 2017 at 4:57 AM, Matthew Brett <matthew.br...@gmail.com> wrote: > Hi, > > On Sun, Jul 23, 2017 at 5:07 PM, Ilhan Polat <ilhanpo...@gmail.com> wrote: >> Ouch, that's from 2012 :( I'll add this thread as a reference to the wiki >> list. >> >> >> On Sun, Jul 23, 2017 at 5:22 PM, Nathan Goldbaum <nathan12...@gmail.com> >> wrote: >>> >>> See >>> https://mail.scipy.org/pipermail/numpy-discussion/2012-August/063589.html >>> and replies in that thread. >>> >>> Quote from an Apple engineer in that thread: >>> >>> "For API outside of POSIX, including GCD and technologies like Accelerate, >>> we do not support usage on both sides of a fork(). For this reason among >>> others, use of fork() without exec is discouraged in general in processes >>> that use layers above POSIX." >>> >>> On Sun, Jul 23, 2017 at 10:16 AM, Ilhan Polat <ilhanpo...@gmail.com> >>> wrote: >>>> >>>> That's probably because I know nothing about the issue, is there any >>>> reference I can read about? >>>> >>>> But in general, please feel free populate new items in the wiki page. >>>> >>>> On Sun, Jul 23, 2017 at 11:15 AM, Nathaniel Smith <n...@pobox.com> wrote: >>>>> >>>>> I've been wishing we'd stop shipping Accelerate for years, because of >>>>> how it breaks multiprocessing – that doesn't seem to be on your list >>>>> yet. >>>>> >>>>> On Sat, Jul 22, 2017 at 3:50 AM, Ilhan Polat <ilhanpo...@gmail.com> >>>>> wrote: >>>>> > A few months ago, I had the innocent intention to wrap LDLt >>>>> > decomposition >>>>> > routines of LAPACK into SciPy but then I am made aware that the >>>>> > minimum >>>>> > required version of LAPACK/BLAS was due to Accelerate framework. Since >>>>> > then >>>>> > I've been following the core SciPy team and others' discussion on this >>>>> > issue. >>>>> > >>>>> > We have been exchanging opinions for quite a while now within various >>>>> > SciPy >>>>> > issues and PRs about the ever-increasing Accelerate-related issues and >>>>> > I've >>>>> > compiled a brief summary about the ongoing discussions to reduce the >>>>> > clutter. >>>>> > >>>>> > First, I would like to kindly invite everyone to contribute and >>>>> > sharpen the >>>>> > cases presented here >>>>> > >>>>> > https://github.com/scipy/scipy/wiki/Dropping-support-for-Accelerate >>>>> > >>>>> > The reason I specifically wanted to post this also in NumPy mailing >>>>> > list is >>>>> > to probe for the situation from the NumPy-Accelerate perspective. Is >>>>> > there >>>>> > any NumPy specific problem that would indirectly effect SciPy should >>>>> > the >>>>> > support for Accelerate is dropped? >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > _______________________________________________ >>>>> > NumPy-Discussion mailing list >>>>> > NumPy-Discussion@python.org >>>>> > https://mail.python.org/mailman/listinfo/numpy-discussion >>>>> > >>>>> >>>>> >>>>> >>>>> -- >>>>> Nathaniel J. Smith -- https://vorpus.org >>>>> _______________________________________________ >>>>> NumPy-Discussion mailing list >>>>> NumPy-Discussion@python.org >>>>> https://mail.python.org/mailman/listinfo/numpy-discussion >>>> >>>> >>>> >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion@python.org >>>> https://mail.python.org/mailman/listinfo/numpy-discussion > > I added some more discussion, and some links to previous discussion on > the mailing list. > > I also pointed to this PR : > https://github.com/MacPython/numpy-wheels/pull/1 - which builds > OpenBLAS wheels for numpy. The same kind of thing would work fine for > Scipy. > > Cheers, > > Matthew > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion@python.org > https://mail.python.org/mailman/listinfo/numpy-discussion -- Nathaniel J. Smith -- https://vorpus.org _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
