Hi, On Sat, Apr 6, 2019 at 11:16 PM Bert JW Regeer <xiste...@0x58.com> wrote: > > > > On Apr 6, 2019, at 23:10, Nathaniel Smith <n...@pobox.com> wrote: > > On Sat, Apr 6, 2019 at 2:14 PM Bert JW Regeer <xiste...@0x58.com> wrote: > > > Hey all, > > You may have seen some hub hub around psycopg2 and no longer shipping binary > wheels by default [1][2] (and in fact using psycopg2-binary if you want > wheels), and I wanted to bring it up here because it demonstrates a problem > area with the current state of packaging in Python: > > There is no good way for a new package to specify that it provides what > another package would provide, and setuptools currently checks that all > distributions are found before running the console scripts (so if a console > script has a setup.py that depends on psycopg2 and you install > psycopg2-binary it fails) [3] > > So currently if you pip install psycopg2-binary and then install a project > that uses psycopg2 as a dependency it will install psycopg2 over top of > psycopg2-binary. > > The author of psycopg2 stopped distributing binaries for psycopg2 because of > issues with the version of SSL that Python was compiled for/what was used by > psycopg2 and it causing all kinds of issues. > > I don't have a proposal or a fix, but this is going to be an issue not just > for psycopg2 but also for other projects that potentially distribute wheels > that are built against a different version of OpenSSL. > > I see two things that should get some thought: > > 1. How to have a package provide for another package (there are keywords but > AFAIK they are currently ignored by pip) > 2. How to handle/deal with shared libraries that are not versioned > > > The psycopg2 authors originally misdiagnosed the problem, and haven't > updated their docs since the problem was diagnosed further, so a lot > of people are confused about this whole psycopg2-binary thing :-( > > There is no problem with shipping openssl in wheels. Lots of projects > do it fine. The reason psycopg2 is having problems is because of an > easily-fixable bug in psycopg2: > > - Old versions of OpenSSL need some annoying configuration applied to > make them thread-safe > - libpq (which psycopg2 uses) normally does this configuration in one way > - the Python ssl module also normally does this configuration in a different > way > - If libpq and the stdlib ssl module are both linked against the > *same* copy of openssl, then they can end up fighting with each other > - So the psycopg2 code has a special hack to unconditionally disable > libpq's thread-safety code, because the psycopg2 developers assumed > that psycopg2 and the stdlib ssl would *always* share the same copy of > openssl, and the stdlib ssl module would take care of the > thread-safety stuff > - Then they started distributing psycopg2 binaries with their own copy > of openssl in them, and of course they got crashes, because they've > turned off thread-safety, and now that they have their own copy of > openssl, no-one else is fixing it for them > > So all they need to do to fix their wheels is either: > > - somehow disable this patch in their wheel builds: > https://github.com/psycopg/psycopg2/commit/a59704cf93e8594dfe59cf12d416e82a816953a4 > > Or else: > > - switch to building their wheels against a newer version of openssl, > since newer versions of openssl are now thread-safe by default (thank > goodness) > > Either way, it's totally fixable, and only the psycopg2 devs can fix it.
I don't work on psychopg2, nor do I understand the threading issues, but I offered to help, current status here: https://github.com/psycopg/psycopg2-wheels/pull/8 Cheers, Matthew -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/B5ODINILHMPXH4LLSOHDQNOKF5YHKANC/
