[Distutils] Re: Installation Instructions for Python?

[Matthew Brett]

Hi,

I'm afraid this isn't the right mailing list for installation instructions
for Python; this is a technical mailing list for discussing the Python
mechanisms for building installable packages for Python libraries.

Instructions on automated installs for Python depend very much on what OS
you are working on - and there are various options in each case.

Best,

Matthew

On Fri, Dec 11, 2020 at 3:15 PM Cort, Brian  wrote:

> Hello,
>
>
>
> We were trying to create an automated installation process.
>
> As part of this, our software deployment team has requested the
> installation instructions, which I was unable to locate.
>
> Can you help by pointing me towards them?
>
>
>
> *Thank you,*
>
>
>
>
>
> *Brian **Brian Cort |*
> * Application Analyst *Kroger Technology: Point of Sale | The Kroger Co.
> office: 503.797.3229 cell: 971.293.6244
>
> For more POS related information, please feel free to see our publicly
> accessible Wiki/Confluence Site:
>
> https://confluence.kroger.com/confluence/x/BySXBQ
>
>
>
> --
>
> This e-mail message, including any attachments, is for the sole use of the
> intended recipient(s) and may contain information that is confidential and
> protected by law from unauthorized disclosure. Any unauthorized review,
> use, disclosure or distribution is prohibited. If you are not the intended
> recipient, please contact the sender by reply e-mail and destroy all copies
> of the original message.
> --
> Distutils-SIG mailing list -- distutils-sig@python.org
> To unsubscribe send an email to distutils-sig-le...@python.org
> https://mail.python.org/mailman3/lists/distutils-sig.python.org/
> Message archived at
> https://mail.python.org/archives/list/distutils-sig@python.org/message/RUNMMTXXNBZYDK7XLLPI6RCHZUIBGSZJ/
>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/XWVZY5ZI7YYQBANU4MKV77ZD7HUEYCQK/

[Distutils] Re: Another travis oddness with multibuild

[Matthew Brett]

Hi,

Yes, that's odd - can you try with multibuild latest?  And then if
that fails, raise an issue on multibuild?

Cheers,

Matthew

On Mon, Feb 10, 2020 at 5:01 PM Robin Becker  wrote:
>
> Not sure if this is just another latest git revision issue; I'm currently 
> using
>
> 0c4a9920fa4635419fcea61f10d6dc0cd50e3639
>
> linux builds are fine
>
>
> I see this building pyRXP-mirror for osx python 2.7 with multibuild
>
> in build_wheel
> > Successfully built pyRXP
> > DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. 
> > Please upgrade your Python as Python 2.7 is no longer maintained. A future 
> > version of pip will drop support for Python 2.7. More details about Python 
> > 2 support in pip, can be found at 
> > https://pip.pypa.io/en/latest/development/release-process/#python-2-support
> > Collecting delocate
> >   Downloading delocate-0.8.0.tar.gz (90 kB) > Requirement already 
> > satisfied: wheel in 
> > /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages
> (from delocate) (0.34.2)
> > Building wheels for collected packages: delocate
> >   Building wheel for delocate (setup.py): started
> >   Building wheel for delocate (setup.py): finished with status 'done'
> >   Created wheel for delocate: filename=delocate-0.8.0-py2-none-any.whl 
> > size=101854 
> > sha256=685b576898d37f65f99a4c3552e8ebfd28aae61803aca183c965cde109b4ff82
> >   Stored in directory: 
> > /Users/travis/Library/Caches/pip/wheels/0c/60/c3/98667c510c1e7da944b945b285d549e1e06c3020a27831f87a
> > Successfully built delocate
> > Installing collected packages: delocate
> >   WARNING: The scripts delocate-addplat, delocate-fuse, delocate-listdeps, 
> > delocate-patch, delocate-path and delocate-wheel are installed in 
> > '/Library/Frameworks/Python.framework/Versions/2.7/bin' which is not on 
> > PATH.
> >   Consider adding this directory to PATH or, if you prefer to suppress this 
> > warning, use --no-warn-script-location.
> > Successfully installed delocate-0.8.0
> > multibuild/osx_utils.sh: line 436: delocate-wheel: command not found
>
> OSX pythons>=3.5 I see success like this
> > Successfully built pyRXP
> > Collecting delocate
> >   Downloading delocate-0.8.0.tar.gz (90 kB)
> > Requirement already satisfied: wheel in ./venv/lib/python3.5/site-packages 
> > (from delocate) (0.34.2)
> > Building wheels for collected packages: delocate
> >   Building wheel for delocate (setup.py): started
> >   Building wheel for delocate (setup.py): finished with status 'done'
> >   Created wheel for delocate: filename=delocate-0.8.0-py3-none-any.whl 
> > size=101854 
> > sha256=f41b49e94615e4acb7e598b3301c04a2191f776bbccde2d02c0ee6076e75042b
> >   Stored in directory: 
> > /Users/travis/Library/Caches/pip/wheels/7e/6e/19/1cb7e3372c9c188b90691e5d690f6eb0b53ab572e5dc7669ed
> > Successfully built delocate
> > Installing collected packages: delocate
> > Successfully installed delocate-0.8.0
> > $ install_run $PLAT
> > Processing ./wheelhouse/pyRXP-2.2.1-cp35-cp35m-macosx_10_6_intel.whl
>
> I guess this is something to do with python 2.7 being built into the virtual 
> machine.
>
> Should I just try with latest or is this an actual problem?
> --
> Robin Becker
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/TTMH5455RIPGYX2HKZBLSAWFVPZCCFXL/

[Distutils] Re: travis failures

[Matthew Brett]

Hi,

Yes, it is an API change in pip 20 - there's a fix in the latest
multibuild devel branch - please update to that ..

https://github.com/matthew-brett/multibuild/issues/298

Cheers,

Matthew

On Wed, Jan 22, 2020 at 4:16 PM Robin Becker  wrote:
>
> I have just seen all my manylinux builds fail at
>
> https://travis-ci.org/MrBitBucket/reportlab-mirror
>
> they all seem to fail in the multibuild install_run phase.
>
> I haven't altered anything related to .travis.yml since the previous run on 
> Jan 17th. All the failures seem to relate to
> a cdn bucket
>
> the runs all end with something like this
>
> Looking in links: 
> https://5cf40426d9f06eb7461d-6fe47d9331aba7cd62fc36c7196769e4.ssl.cf2.rackcdn.com
> 1483Collecting pillow
> 1484  Downloading Pillow-6.2.2-cp27-cp27m-manylinux1_i686.whl (2.0 MB)
> 1485Installing collected packages: pillow
> 1486Successfully installed pillow-6.2.2
> 1487DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. 
> Please upgrade your Python as Python 2.7
> is no longer maintained. A future version of pip will drop support for Python 
> 2.7. More details about Python 2 support
> in pip, can be found at 
> https://pip.pypa.io/en/latest/development/release-process/#python-2-support
> 1488ERROR: You must give at least one requirement to install (maybe you meant 
> "pip install
> https://5cf40426d9f06eb7461d-6fe47d9331aba7cd62fc36c7196769e4.ssl.cf2.rackcdn.com;?)
>
> but I cannot find that in my .yml or the travis config file which I use.
>
> Any idea what could be causing this?
> --
> Robin Becker
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/XMACHC3CE72JPN4PCBBD3Y3S6ZICRDM2/

[Distutils] Re: Linux binary wheels?

[Matthew Brett]

Hi,

Yes, here you are trying to upload a generic Linux wheel.   PyPI
refuses, because there's no way Pip can tell if the generic Linux
wheel will work for any particular Linux distribution.For example,
you could build a LInux wheel on Debian 10, but someone on Redat 6
could try to install it - chances are that the installed package will
be badly broken.This is the reason for the Manylinux standard - it
gives Pip a standard to check against, to see whether the Manylinux
wheel will work on the installing platform.

Cheers,

Matthew

On Wed, Aug 21, 2019 at 3:09 PM Dan Stromberg  wrote:
>
>
> It sounds pretty similar, but I'm still getting:
> $ twine upload --verbose --repository-url https://test.pypi.org/legacy/ dist/*
> below cmd output started 2019 Wed Aug 21 07:05:31 AM PDT
> Enter your username: dstromberg
> Enter your password:
> Uploading distributions to https://test.pypi.org/legacy/
> Uploading treap-1.39-cp36-cp36m-linux_x86_64.whl
> 100%|██|
>  427k/427k [00:01<00:00, 306kB/s]
> Content received from server:
> 
>  
>   400 Binary wheel 'treap-1.39-cp36-cp36m-linux_x86_64.whl' has an 
> unsupported platform tag 'linux_x86_64'.
>  
>  
>   400 Binary wheel 'treap-1.39-cp36-cp36m-linux_x86_64.whl' has an 
> unsupported platform tag 'linux_x86_64'.
>   The server could not comply with the request since it is either malformed 
> or otherwise incorrect.
> Binary wheel treap-1.39-cp36-cp36m-linux_x86_64.whl has an 
> unsupported platform tag linux_x86_64.
>
>
>  
> 
> HTTPError: 400 Client Error: Binary wheel 
> 'treap-1.39-cp36-cp36m-linux_x86_64.whl' has an unsupported platform tag 
> 'linux_x86_64'. for url: https://test.pypi.org/legacy/
>
> If a better way is available now, the error message probably should point the 
> user at it.
>
> ?
>
> Thanks!
>
> On Mon, Aug 19, 2019 at 6:12 PM Wes Turner  wrote:
>>
>> How does this proposal differ from manylinux2010?
>>
>> https://github.com/pypa/manylinux/blob/master/README.rst#example
>>
>> PEP 513: manylinux1
>> https://www.python.org/dev/peps/pep-0513/
>>
>> PEP 571: The manylinux2010 Platform Tag (latest, as of 2019)
>> https://www.python.org/dev/peps/pep-0571/
>>
>>
>>
>> On Monday, August 19, 2019, Dan Stromberg  wrote:
>>>
>>> Hi folks.
>>> I have a pair of ideas about Linux binary wheels, which are currently (I 
>>> heard) unsupported.
>>> It seems like it should be possible to support Linux binary wheels using 
>>> one or both of these technologies:
>>> * https://build.opensuse.org/ is a service that builds packages for a 
>>> variety of Linuxes
>>> * Docker could be used to automate the building of wheels for a handful of 
>>> Linuxes with minimal dependencies.  It seems like if you get 
>>> Debian/Ubuntu/Mint, Fedora/CentOS, openSUSE and perhaps one or two others, 
>>> that would cover almost all Linuxes and Linux users.
>>>
>>> I'm up to my hears in commitments already, but I sincerely someone will 
>>> grab onto one or both of these possibilities and run with them.
>>> Thanks for reading.
>>>
> --
> Distutils-SIG mailing list -- distutils-sig@python.org
> To unsubscribe send an email to distutils-sig-le...@python.org
> https://mail.python.org/mailman3/lists/distutils-sig.python.org/
> Message archived at 
> https://mail.python.org/archives/list/distutils-sig@python.org/message/RSJ57E343I7NVB6QDFFBD4QNTHRVAYOT/
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/AJHTJAP5OCX6JRIUH3PGJTX5YQUGTMEG/

[Distutils] Re: Linux binary wheels?

[Matthew Brett]

Hi,

On Tue, Aug 20, 2019 at 11:10 AM Brian Skinn  wrote:
>
>
>
> On Tue, Aug 20, 2019, 06:05 Matthew Brett  wrote:
>>
>> Hi,
>
>
>> See the links that Wes posted for more details, or try pip installing
>> Numpy and Scipy in a new virtualenv, and see what happens.
>
>
> Probably needs to:
>
>pip install --only-binary :all: numpy scipy

Pip doesn't build from source by default - here's the output from my
machine just now:

 $ pip install numpy scipy
Collecting numpy
  Downloading 
https://files.pythonhosted.org/packages/4a/2e/cf0a2fea6d97604a0e058e804b50d589c31b360b317be9f5c126b22a560e/numpy-1.17.0-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
(14.9MB)
 || 15.0MB 38.9MB/s
Collecting scipy
  Downloading 
https://files.pythonhosted.org/packages/c5/40/f73bc951b060ba143f2092a3a2dd755f5ef2b52376fc3f4d297942aef04b/scipy-1.3.1-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
(27.6MB)
 || 27.6MB 35.8MB/s
Installing collected packages: numpy, scipy
Successfully installed numpy-1.17.0 scipy-1.3.1

Cheers,

Mattthew
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/UX4LUBUGQXYID3HB5FADEVR4ACGKFHL6/

[Distutils] Re: Linux binary wheels?

[Matthew Brett]

Hi,

On Tue, Aug 20, 2019 at 1:23 AM Dan Stromberg  wrote:
>
> Hi folks.
> I have a pair of ideas about Linux binary wheels, which are currently (I 
> heard) unsupported.

Your information is wrong, I'm afraid - Manylinux wheels have been
standard, and widely used, for several years now.   Unless you meant
wheels for non-Intel platforms, in which case, please do say more
about you need.

See the links that Wes posted for more details, or try pip installing
Numpy and Scipy in a new virtualenv, and see what happens.

Cheers,

Matthew
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/HXQWSOTX5V4VD2NX573JWX7JRT2Y4EYJ/

[Distutils] Re: psycopg2/psycopg2-binary

[Matthew Brett]

Hi,

On Sat, Apr 6, 2019 at 11:16 PM Bert JW Regeer  wrote:
>
>
>
> On Apr 6, 2019, at 23:10, Nathaniel Smith  wrote:
>
> On Sat, Apr 6, 2019 at 2:14 PM Bert JW Regeer  wrote:
>
>
> Hey all,
>
> You may have seen some hub hub around psycopg2 and no longer shipping binary 
> wheels by default [1][2] (and in fact using psycopg2-binary if you want 
> wheels), and I wanted to bring it up here because it demonstrates a problem 
> area with the current state of packaging in Python:
>
> There is no good way for a new package to specify that it provides what 
> another package would provide, and setuptools currently checks that all 
> distributions are found before running the console scripts (so if a console 
> script has a setup.py that depends on psycopg2 and you install 
> psycopg2-binary it fails) [3]
>
> So currently if you pip install psycopg2-binary and then install a project 
> that uses psycopg2 as a dependency it will install psycopg2 over top of 
> psycopg2-binary.
>
> The author of psycopg2 stopped distributing binaries for psycopg2 because of 
> issues with the version of SSL that Python was compiled for/what was used by 
> psycopg2 and it causing all kinds of issues.
>
> I don't have a proposal or a fix, but this is going to be an issue not just 
> for psycopg2 but also for other projects that potentially distribute wheels 
> that are built against a different version of OpenSSL.
>
> I see two things that should get some thought:
>
> 1. How to have a package provide for another package (there are keywords but 
> AFAIK they are currently ignored by pip)
> 2. How to handle/deal with shared libraries that are not versioned
>
>
> The psycopg2 authors originally misdiagnosed the problem, and haven't
> updated their docs since the problem was diagnosed further, so a lot
> of people are confused about this whole psycopg2-binary thing :-(
>
> There is no problem with shipping openssl in wheels. Lots of projects
> do it fine. The reason psycopg2 is having problems is because of an
> easily-fixable bug in psycopg2:
>
> - Old versions of OpenSSL need some annoying configuration applied to
> make them thread-safe
> - libpq (which psycopg2 uses) normally does this configuration in one way
> - the Python ssl module also normally does this configuration in a different 
> way
> - If libpq and the stdlib ssl module are both linked against the
> *same* copy of openssl, then they can end up fighting with each other
> - So the psycopg2 code has a special hack to unconditionally disable
> libpq's thread-safety code, because the psycopg2 developers assumed
> that psycopg2 and the stdlib ssl would *always* share the same copy of
> openssl, and the stdlib ssl module would take care of the
> thread-safety stuff
> - Then they started distributing psycopg2 binaries with their own copy
> of openssl in them, and of course they got crashes, because they've
> turned off thread-safety, and now that they have their own copy of
> openssl, no-one else is fixing it for them
>
> So all they need to do to fix their wheels is either:
>
> - somehow disable this patch in their wheel builds:
> https://github.com/psycopg/psycopg2/commit/a59704cf93e8594dfe59cf12d416e82a816953a4
>
> Or else:
>
> - switch to building their wheels against a newer version of openssl,
> since newer versions of openssl are now thread-safe by default (thank
> goodness)
>
> Either way, it's totally fixable, and only the psycopg2 devs can fix it.

I don't work on psychopg2, nor do I understand the threading issues,
but I offered to help, current status here:

https://github.com/psycopg/psycopg2-wheels/pull/8

Cheers,

Matthew
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/B5ODINILHMPXH4LLSOHDQNOKF5YHKANC/

[Distutils] Re: Depending on external library

[Matthew Brett]

Hi,

On Fri, Feb 15, 2019 at 6:15 PM Jeroen Demeyer  wrote:
>
> On 2019-01-22 13:04, Robert T. McGibbon wrote:
> > It's necessary to bundle the external library with the wheel.
> >
> > On linux, this is what the manylinux tag is all about (see PEP 513 &
> > 571) and the auditwheel tool (https://github.com/pypa/auditwheel) will
> > bundle the library into the wheel.
> >
> > On OS X and Windows, I'm less familiar with the toolchain but the same
> > principle applies.
>
> Thanks for your reply. Unfortunately as you say, it only applies to Linux.
>
> So how do projects in practice deal with this in a cross-platform way?
> Does every package need to reinvent the "wheel"?

Delocate [1] deals with the same problem on macOS  - see my Multibuild
frameworks [2] for some shared tooling between the two.  It's fairly
widely used (numpy, scipy, matplotlib, etc).

I think the same general approach would be possible with Windows - see
Nathaniel Smith's library at [3] - but I don't think anyone has done
the work to package that into a wheel-building tool.  Most people are
building Windows wheels with static linking, I believe.

Nathaniel Smith proposed a specification for packaging binary
libraries as wheels [4], but I think he isn't working on that now, and
it hasn't progressed.  Nathaniel - is that right?

Cheers,

Matthew

[1] https://github.com/matthew-brett/delocate
[2] https://github.com/matthew-brett/multibuild
[3] https://github.com/njsmith/machomachomangler
[4] 
https://github.com/njsmith/wheel-builders/blob/pynativelib-proposal/pynativelib-proposal.rst
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/53MDA4EEEQHE3EXWTIMP6FLAV2R5PFUV/

[Distutils] Re: Idea: perennial manylinux tag

[Matthew Brett]

Hi,
On Sat, Dec 1, 2018 at 5:18 PM Thomas Kluyver  wrote:
>
> Thanks Nathaniel for the explanation.
>
> On Sat, Dec 1, 2018, at 4:39 AM, Nathaniel Smith wrote:
> > So the proposal here is to refactor the spec to match how this
> > actually works: the official definition of a manylinux_${glibc
> > version}_${arch} wheel would be "I promise this wheel will work on any
> > Linux system with glibc >=${glibc version} and an ${arch} processor".
>
> I'm still a bit unsure how this works with the other libraries specified in 
> PEP 571 (glib, libXrender, etc.). Would they be entirely dropped from a 
> hypothetical manylinux_2_20, so wheels need to bundle everything apart from 
> glibc itself? Or is it reasonable to assume that any system built with glibc 
> has certain other libraries available? And is there any need to specify 
> versions of these libraries, or is e.g. libX11.so.6 sticking around forever?

I think this is the key point.

For Mac - Apple has already done the specification work for us, with
its MACOSX_DEPLOYMENT_TARGET specifiers.   These versions, such as
'10.6' specify compatible versions for all the system libraries.

I don't know Windows well, but I suppose that the equivalent APIs to
the PEP 571 libraries are stable across many Windows versions, and
it's standard Windows practice to compile against old and stable APIs.

Cheers,

Matthew
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/NHYJWB7QYI4WVBCQJ7L3AVEATANP3QUA/

[Distutils] Re: Intent to add a "Packaging" category to discuss.python.org

[Matthew Brett]

Hi,
On Sat, Nov 3, 2018 at 11:30 AM Paul Moore  wrote:
>
> On Sat, 3 Nov 2018 at 01:27, Donald Stufft  wrote:
> >
> > Just an FYI, I’m going to ask the discuss.python.org admins to add a 
> > Packaging category to that discourse instance for the discussion of 
> > packaging in Python, basically as a sister discussion location to 
> > distutils-sig. I plan to primarily participate and centralize my own 
> > personal discussions there, although others are free to continue to discuss 
> > wherever they would like (although I think it would be great if folks would 
> > all jump over there as well!).
> >
> > I’m not going to request any sub categories at this time, both because I 
> > don’t think our traffic is high enough to warrant it, and also because I 
> > think that tagging will be a more useful mechanism for us since we can tag 
> > things with relevant projects or whatever (or not tag things at all) and 
> > topics can have multiple tags.
>
> For now, can I ask that any discussions that need my participation
> post here. I don't currently have the bandwidth to learn/follow
> Discourse, so I'm extremely likely to miss things that are posted on
> there. Specifically, any discussions around packaging standards that
> need a decision from me, or need my input, should take place on this
> list.

You won't miss me much, I'm sure, but I'm also overwhelmed with email,
and I really don't want to have another channel to distract me.  So, I
guess I'm hoping that y'all will bring some summary back to the list
for those of us, who are not following Discourse.

Cheers,

Matthew
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/UQ762M5OA4KUNTDUNG3FFNYCNP2TX7J3/

[Distutils] Re: Adopting virtualenv package maintenance

[Matthew Brett]

Hi,

On Thu, Sep 6, 2018 at 9:42 PM, Alex Becker  wrote:
> Another +1 to the utility of a maintainer. I am also working on package
> management and have found that venv is not a full replacement for
> virtualenv--for example I don't believe the environment can be entered
> programatically, while virtualenv provides activate_this.py which can be
> exec'd. I'm sure there are many other limitations, so I don't think python
> can give up on virtualenv soon.

Just to add - I use virtualenv often, and I would be happy to know it
is maintained.   On reflection, the reason I am not using venv is
because a) I still need to test on Python 2.7 and b) I like
`virtualenvwrapper`.   I suppose someone could rewrite
virtualenvwrapper to use venv instead, but a) is only going to stop
being a requirement sometime after 2020.

Cheers,

Matthew
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/2ODF6E3UJLJXK7A5YQHJ7UIFQYUIPOLI/

[Distutils]Re: [Wheel-builders] [ANN] Python 3.7 now available in the manylinux docker images

[Matthew Brett]

Hi,


On Mon, Jun 25, 2018 at 8:55 AM, Robin Becker  wrote:
> On 21/06/2018 19:18, Matthew Brett wrote:
>>
>> Hi,
>>
>> On Thu, Jun 21, 2018 at 12:45 PM, Robin Becker 
>> wrote:
>>>
>>> On 15/06/2018 23:42, Nathaniel Smith wrote:
>>>>
>>>>
> ..
>>>
>>> looking in side the wheels it looks like the .so files have been slimmed.
>>> Does anyone know if that is just because they no longer include debugging
>>> information etc etc?
>>
>>
>> If you haven't built in a while, this could well be the result of
>> merging https://github.com/matthew-brett/multibuild/pull/140 - which
>> introduced stripped Linux binaries.
>>
> thanks very much. Looking in configure_build.sh it seems I can get a
> debugging build by exporting STRIP_FLAGS='' or is there a another way?

Right - simplest would be to put STRIP_FLAGS=" " (notice space inside
quotes) in your `env_vars.sh` file, to disable the stripping.  This
gets sourced before `configure_build.sh`.

Cheers,

Matthew
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/TASBUZLHVWFL3T6SYQCOH6A4DNBGW6XB/

[Distutils] Re: Excluding specific Python modules from wheels

[Matthew Brett]

Hi,

On Wed, May 30, 2018 at 10:29 PM,   wrote:
> I am currently working on implementing a "wheel repack" command which lets
> you unpack a wheel, modify it and then repack it again while keeping the
> wheel RECORD consistent.
> I think this is something that would solve your use case.

I wrote something similar a while back, of form:

with InWheel(original_wheel_fname, output_wheel_fname):
# You are now in the root directory of the unpacked wheel.
# Do what you will.  The resulting files get packed up into the
output wheel.
# The record file adapted accordingly.

https://github.com/matthew-brett/delocate/blob/master/delocate/wheeltools.py

Cheers,

Matthew
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/D2AOY5YUOOTAJSN3DZKIXXQKYQKNGW2P/

Re: [Distutils] Announcement: Pip 10 is coming, and will move all internal APIs

[Matthew Brett]

Hi,

On Sun, Oct 22, 2017 at 8:52 AM, Elvis Stansvik
 wrote:
> 2017-10-21 14:34 GMT+02:00 Paul Moore :
>> On 21 October 2017 at 12:15, Nick Coghlan  wrote:
>>> (Note: this is entirely speculative, and I have no idea how hard it would
>>> be, so please read it as the question it's intended to be)
>>
>> No problem - I don't know myself how hard some of this would be, either ;-)
>>
>>> Do you know if there any key APIs (like installation) that could be turned
>>> into wrappers around pip CLI calls in order to mitigate some of the impact?
>>
>> The obvious one is pip.main(). That's the one a lot of people use, but
>> it's easily replaceable by a simple subprocess call. That's actually
>> one of the reasons this was so frustrating to us - the bug reports we
>> got were often from people doing things they didn't need to, that they
>> could handle trivially via a supported approach.
>>
>> Otherwise, no. We've had little or no feedback on the tracker from
>> people using more complex internals, so our working assumption has
>> been there's very little that can't be handled via the CLI or existing
>> packages. Feedback so far from this mail hints that maybe we were
>> wrong, but it's still hard to know if it's one or two key projects, or
>> a whole range of people that we've yet to hear from. I'm pretty sure,
>> for example, that pipenv uses internals, either directly or via one of
>> their dependencies, but we've not seen any feedback from them yet.
>
> Another one that immediately comes to mind is pip-tools [1], which I
> think is quite widely used.
>
> But I just checked, and they filed a "check out how to deal with pip
> 10" issue two days ago [2] (I'm guessing in response to this thread).

Now pip 10 is out, of course, I discover that I've lost the
implementation of `get_supported` in pip.pep425tags.  It's my fault
for not remembering I had used it.

Is the suggestion to use the `_internal` import, or carry a copy of
the pep425tags code myself, that I have to keep up to date with the
internal pip copy?  That would also involve me copying the `glibc`
part of the code.  I see that the `wheel` package has an old copy of
that code too, which doesn't deal with manylinux wheels.You
probably saw that `pip-tools` ended up vendoring the whole of pip9
[1].

Cheers,

Matthew

[1] https://github.com/jazzband/pip-tools/tree/master/piptools/_vendored/pip
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Need how to install 3rd party python modules

[Matthew Brett]

Hi,

Brett is just saying that you should first open a "cmd" shell, and, at
the cmd prompt, type:

pip install openpyxl

Your problem (which is relatively common) is that you were typing the
command at the Python prompt, instead of the cmd prompt.

Best,

Matthew


On Tue, Mar 27, 2018 at 6:04 PM, 森平 直樹  wrote:
> Brett,
>
>
>
> Thank you for responding me.
>
> Could you tell me the e-mail address of the python-tutor ?
>
> And if you have something even a little to be a hint to resolve this syntax
> error, it is appreciated to teach me.
>
>
>
> Best Regards,
>
> -
> Naoki Morihira
> TEL: 01181-90-6460-6265
> -
>
>
>
> 
> 差出人: Brett Cannon 
> 送信日時: Tuesday, March 27, 2018 9:44:57 AM
> 宛先: ?? ??
> CC: distutils-sig@python.org
> 件名: Re: [Distutils] Need how to install 3rd party python modules
>
> For general help to get started with Python I would recommend the
> python-tutor mailing list; this one is for discussing the development of how
> to package things in Python.
>
> But in your specific case you typed the command in Python's REPL and not at
> the terminal. For further help please ask on python-tutor.
>
> On Tue, 27 Mar 2018 at 09:42 ?? ??  wrote:
>>
>> I would like to install openpyxl, and
>>
>> I installed Python shown as follows:
>>
>> Python 3.6.4 (v3.6.4:d48eceb, Dec 19 2017, 06:04:45) [MSC v.1900 32 bit
>> (Intel)] on win32
>>
>>
>>
>> But as the following situation shows, I cannot succeed installing
>> openpyxl.
>>
>>
>>
>> >>> pip install openpyxl
>>
>>   File "", line 1
>>
>> pip install openpyxl
>>
>>   ^
>>
>> SyntaxError: invalid syntax
>>
>>
>>
>> How can I succeed installing openpyxl ?
>>
>>
>>
>> Best Regards,
>>
>> -
>> Naoki Morihira
>> TEL: 01181-90-6460-6265
>> -
>>
>>
>>
>> ___
>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] pypi.python Error 403 Forbidden

[Matthew Brett]

Hi,

On Tue, Feb 13, 2018 at 8:19 PM, Heiko L.  wrote:
> hallo,
>
> I try to install a software from pypi.python.org and receive following errmsg:
> urllib2.HTTPError: HTTP Error 403: Forbidden
>
> and
>
> $ wget 
> http://pypi.python.org/packages/2.7/s/setuptools/setuptools-0.6c11-py2.7.egg
> HTTP request sent, awaiting response... 403 SSL is required
>
>
> After a few hours I found the following article:
>  https://mail.python.org/pipermail/distutils-sig/2017-October/031712.html
>   ...you can no longer access /simple/ and /packages/ over HTTP and you will 
> have to directly go to HTTPS
>
> - It is true?
> - Is this the right place if I have any questions?

Can you use pip to install your software instead?What are you
trying to install, exactly?  Is it just this old version of
setuptools?  Why do you need that?

Cheers,

Matthew
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] draft PEP: manylinux2

[Matthew Brett]

On Sun, Feb 11, 2018 at 7:53 AM, Alex Walters  wrote:
> Just out of curiosity, I did a little experiment.  I explained this thread
> to my mother.  My mother is a wonderful woman, but she wouldn't know a byte
> from a bite.  I explained it as follows:
>
> "There is a tool that can make software run on a lot of different computers,
> but only if you build it for an ancient computer.  The tool is a little
> complicated - you have to learn how to get it and to use it with any
> success.  The people who make it are considering changing the way they name
> it.  The new naming scheme is the bare minimum year the computer running the
> code can be from.

I think the problem is that the whole discussion turns on whether we
should care about the fact that it's more complicated than the last
sentence would suggest.

Cheers,

Matthew
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPI Limits of Wheel Size

[Matthew Brett]

Hi,

On Mon, Nov 13, 2017 at 10:07 AM, Phil Thompson
 wrote:
> I am hitting a limit on the size of a wheel when uploading to PyPI - the 
> wheel is about 100M. When this has happened in the past I have contacted 
> Donald who has been kind enough to increase the limit (I believe it is a 
> per-project limit).
>
> Currently where should I make this request?

I'm hoping / guessing that this is a PyQt wheel?  That would be a huge help ...

Cheers,

Matthew
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Announcement: Pip 10 is coming, and will move all internal APIs

[Matthew Brett]

Hi,

On Fri, Oct 20, 2017 at 2:22 PM, Paul Moore  wrote:
> We're in the process of starting to plan for a release of pip (the
> long-awaited pip 10). We're likely still a month or two away from a
> release, but now is the time for people to start ensuring that
> everything works for them. One key change in the new version will be
> that all of the internal APIs of pip will no longer be available, so
> any code that currently calls functions in the "pip" namespace will
> break. Calling pip's internal APIs has never been supported, and
> always carried a risk of such breakage, so projects doing so should,
> in theory, be prepared for such things. However, reality is not always
> that simple, and we are aware that people will need time to deal with
> the implications.
>
> Just in case it's not clear, simply finding where the internal APIs
> have moved to and calling them under the new names is *not* what
> people should do. We can't stop people calling the internal APIs,
> obviously, but the idea of this change is to give people the incentive
> to find a supported approach, not just to annoy people who are doing
> things we don't want them to ;-)
>
> So please - if you're calling pip's internals in your code, take the
> opportunity *now* to check out the in-development version of pip, and
> ensure your project will still work when pip 10 is released.
>
> And many thanks to anyone else who helps by testing out the new
> version, as well :-)

Thanks for the heads-up.

Will y'all be doing a PyPI pre-release so we can test with `pip
install --pre -U pip`?

Cheers,

Matthew
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Install psutil without gcc?

[Matthew Brett]

Hi,

On Fri, Sep 8, 2017 at 2:42 PM, Wes Turner  wrote:
> Is there a reason that psutil cannot be built as a manylinux1 wheel?
>
> - manylinux1_x86_64
> - manylinux1_i686

Thomas can correct me if I'm wrong, but I think he was looking for an
easy automated way to do this.

Cheers,

Matthew
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Install psutil without gcc?

[Matthew Brett]

Hi,

On Fri, Sep 8, 2017 at 1:54 PM, Thomas Güttler
<guettl...@thomas-guettler.de> wrote:
> It would be great, if I could install psutil without the need for a gcc on
> linux.
>
> The maintainer (Giampaolo Rodola) is interested, but the roadmap is not
> clear yet.
>
> He said:
>
>> It's something which needs to be investigated. A quick Google searched
>> brought me here:
>> https://github.com/joerick/cibuildwheel
>> ...so it appears it's definitively possible.
>
> Is this the current state of the art, or is there a better way?

I haven't looked at CIBuildwheel in much detail, but my impression is
that it will work well for simple wheels, but I didn't investigate far
to see how well it would cope for more substantial builds.

My own project for the same thing is somewhat less streamlined, but is
relatively flexible:

https://github.com/matthew-brett/multibuild

I did a lot of wheel-build bootstrapping. so there are a lot of
projects that use it - numpy, scipy, matplotlib, pandas, Cython, h5py,
Pillow, scikit-learn, scikit-image, astropy ...  Of course that
density is largely historical, but it does at least mean that the
project is battle tested and relatively well maintained.   I'm happy
to help you get set up.

You might also look at Scikit-Build -
https://github.com/scikit-build/scikit-build - I believe that project
is particularly good if you are using CMake.

By the way, you might prefer the wheel-builders list for this
discussion - https://mail.python.org/mailman/listinfo/wheel-builders

Cheers,

Matthew
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Matthew Brett]

Hi,

On Tue, Aug 29, 2017 at 8:30 PM, xoviat  wrote:
>> For me, using NotImplemented is a misuse of the singleton since I know
>> what it's meant to be used for (and so I cringe every time I hear it brought
>> up as a solution).
>
> I hate to reiterate too much, but I think that you are correct here:
> NotImplemented is not Pythonic. It's a complete hack that I would think
> people would like to quarantine to the limited use case of binary
> operations. Even with the poorly contextualized email that Guido received, I
> think it was clear that he thought it wasn't appropriate outside of that use
> case either (I don't think better context would have changed that).
>
> Personally, I was willing to overlook that for the sake of a speedy
> resolution. But reading the email chain, it seems that only Thomas and
> Daniel are currently in favor of using NotImplemented, which is a shift from
> the earlier situation. So now it seems that NotImplemented is not likely to
> be used.

Just for the record, I thought Nathaniel's NotImplemented suggestion
was a good one, and his explanation of why, was clear and convincing,
but he's already accepted the alternative, of a custom error, so I
don't think there's much point on going back over that.

Also just for the record, I'd like to warmly thank y'all for your
great patience in this long discussion.

Cheers,

Matthew
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] manylinux1 testing

[Matthew Brett]

Hi,

On Mon, Jun 5, 2017 at 12:11 PM, Robin Becker <ro...@reportlab.com> wrote:
> On 05/06/2017 10:34, Matthew Brett wrote:
>>
>> Hi,
>>
> ...
>>
>>
>> You could do that, they will test whether the wheels run a minimal
>> case in terms of versions and dependencies.  You may hit
>> inconveniences building packages, because the toolchains are rather
>> old.
>>
> I assumed that everything I need would be in the wheels I have built. The
> manylinux docker needs to be able to build pythons anyhow so I guess the
> tools are good enough for that.

Sure - the building I was think of was for any dependencies where you
didn't have a built wheel.

You'll likely be able to get by with the tools on the Manylinux docker
image, but you might have to pay attention to stuff like which version
of cmake you have and so on.

Cheers,

Matthew
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] manylinux1 testing

[Matthew Brett]

Hi,

On Mon, Jun 5, 2017 at 10:00 AM, Robin Becker <ro...@reportlab.com> wrote:
> Are there docker images suitable for testing manylinux packages. I am
> guessing the manylinux1 images might be suitable, provided I drop all
> building of libraries inside them and then just use the pythons I want to
> test against.
>
> Reason I'm asking is that I noticed that I was building images where the
> final grafting didn't seem to be happening I assume something must have been
> wrong with the repair_wheehouse command.
>
> My manylinux1 images were 4 weeks old. The situation improved after I
> updated today.
>
> However, I have been testing these 'bad' wheels and they all appeared to
> work. Problem is that I'm only bundling freetype, libpng and libz and I
> guess my test systems all have those.
>
> So is it reasonable for me to use the manylinux1 images as a null linux so I
> can pip load my stuff from the local file system and test it all runs on a
> system with almost no libraries built in?

You could do that, they will test whether the wheels run a minimal
case in terms of versions and dependencies.  You may hit
inconveniences building packages, because the toolchains are rather
old.

Alternatively you could use a bare but more recent distribution.  For
example, my multibuild framework for manylinux testing uses manylinux
containers to build with, and bare Ubuntu Trusty images for testing:

https://github.com/matthew-brett/multibuild
https://hub.docker.com/r/matthewbrett/trusty

Cheers,

Matthew
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] installing pip or at least modules on mac

[Matthew Brett]

Hi,

On Sun, May 21, 2017 at 12:01 AM, Sébastien Turrel
 wrote:
> iMac-de-Sebastien:3.5 sebastienturrel$ cd bin
> iMac-de-Sebastien:bin sebastienturrel$ ls
> 2to3pydoc3python3.5-config
> 2to3-3.5pydoc3.5python3.5m
> easy_install-3.5python3python3.5m-config
> idle3python3-32pyvenv
> idle3.5python3-configpyvenv-3.5
> pip3python3.5
> pip3.5python3.5-32
> iMac-de-Sebastien:bin sebastienturrel$ pip3.5
>
> Usage:
>   pip  [options]
>
> Commands:
>   install Install packages.
>   downloadDownload packages.
>   uninstall   Uninstall packages.
>   freeze  Output installed packages in requirements
> format.
>   listList installed packages.
>   showShow information about installed packages.
>   check   Verify installed packages have compatible
> dependencies.
>   search  Search PyPI for packages.
>   wheel   Build wheels from your requirements.
>   hashCompute hashes of package archives.
>   completion  A helper command used for command completion.
>   helpShow help for commands.
>
> General Options:
>   -h, --help  Show help.
>   --isolated  Run pip in an isolated mode, ignoring
>   environment variables and user configuration.
>   -v, --verbose   Give more output. Option is additive, and can
> be
>   used up to 3 times.
>   -V, --version   Show version and exit.
>   -q, --quiet Give less output. Option is additive, and can
> be
>   used up to 3 times (corresponding to WARNING,
>   ERROR, and CRITICAL logging levels).
>   --log Path to a verbose appending log.
>   --proxy  Specify a proxy in the form
>   [user:passwd@]proxy.server:port.
>   --retries  Maximum number of retries each connection
> should
>   attempt (default 5 times).
>   --timeout  Set the socket timeout (default 15 seconds).
>   --exists-action Default action when a path already exists:
>   (s)witch, (i)gnore, (w)ipe, (b)ackup, (a)bort.
>   --trusted-hostMark this host as trusted, even though it does
>   not have valid or any HTTPS.
>   --certPath to alternate CA bundle.
>   --client-cert Path to SSL client certificate, a single file
>   containing the private key and the certificate
>   in PEM format.
>   --cache-dirStore the cache data in .
>   --no-cache-dir  Disable the cache.
>   --disable-pip-version-check
>   Don't periodically check PyPI to determine
>   whether a new version of pip is available for
>   download. Implied with --no-index.
> iMac-de-Sebastien:bin sebastienturrel$ install PyPDF2

Here you're running the 'install' system command.  What you want is to
run the pip 'install' subcommand:

On my machine:

$ pip install PyPDF2
Collecting PyPDF2
  Downloading PyPDF2-1.26.0.tar.gz (77kB)
100% || 81kB 824kB/s
Installing collected packages: PyPDF2
  Running setup.py install for PyPDF2 ... done
Successfully installed PyPDF2-1.26.0

Best,

Matthew
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] The Python Packaging Ecosystem (of Nick) Support for other programming languages

[Matthew Brett]

Hi,

On Sat, Apr 8, 2017 at 6:55 AM, Nick Coghlan  wrote:
> On 7 April 2017 at 15:26, Thomas Güttler  wrote:
>> Why is having blue sky ideas rude? AFAIK the word "rude" means "offensively 
>> impolite or bad-mannered."
>
> Ideas are easy to come by - we have no shortage of them.

Bad ideas are surely easy to come by, but, at least in my experience,
it often takes a bit of discussion to work out which ones these are.

> Hearing from folks that say "I'm working on a project to improve X,
> can you give me some advice?" is generally wonderful, but "Someone
> (else) should totally build this thing that I wish existed" is
> typically just noise, and
> "I have never personally done anything for any of you, but I want you
> all to imagine you work for me and have to work on the things I care
> about" is extraordinarily self-entitled behaviour.

Wishing to avoid this criticism I humbly submit the pip / manylinux /
macOS packaging work I do for the scientific Python stack.

That done, as you imply, no-one can force us volunteers to do any
particular piece of work, so, although self-entitlement may be
unattractive, it's not particularly threatening.  Of course,
sometimes, people who aren't doing anything at present, suggest ideas
that are useful, and with the right encouragement, actually do start
to help.  I've certainly seen that happen with my other developer hats
on.

Cheers,

Matthew
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] latest setuptools appears to require six in a breaking way

[Matthew Brett]

Hi,

On Wed, Jan 25, 2017 at 4:52 PM, Donald Stufft  wrote:
>
> On Jan 25, 2017, at 12:27 PM, Eric Brunson  wrote:
>
> It wasn't until recently the I realized how quickly releases to setuptools
> and pip are being made, starting back in mid Dec when much of our dependency
> resolution started failing.  There were three releases in the past two days.
> Four major and 22 minor releases in the past two months.  While I applaud
> the speed of development and the improvement in these tools, don't you feel
> that breaking changes should be advertised better before release or perhaps
> we should slow down the cadence for release?
>
> I think an expectation that every setuptools user in the community start
> their day by checking to see if there was a release in the past 24 hours is
> a little unreasonable.  I've spent a dozen hours since 32.0.0 resolving
> breakage in my own projects and assisting other developers in my org with
> their setuptools issues, all the while pushing setuptools as the best
> practice to do development and distribution.  Is this period of breaking
> changes a short term thing that we just have to tough out for a few more
> weeks?
>
> Thanks,
> e.
>
>
>
> I don’t believe that pip is really releasing that quickly. We generally make
> 1-2 “major” versions a year that include breaking changes, 2-4 “minor”
> releases a year that add new features, and 6-10 patch releases that fix
> bugs. To me that feels like a pretty decent pace of balancing not breaking
> people and getting new changes into people’s hands and getting rid of broken
> or less optimal parts of the code.
>
> Now, setuptools is releasing faster than pip is and whether that’s a good
> thing or not I don’t know. That’s a question for Jason largely :)

It sounds like we need to get some continuous integration to bear on
this problem.

How about the following suggestion: before new releases, make release
candidates for pip, wheel, setuptools and virtualenv, upload to pypi,
announce here.

Then we the dependent projects can have a continuous integration entry
which tests our normal pip install with the --pre versions of these
packages, to screen for trouble.   I think that would catch many of
the problems, and it doesn't seem like it would be much work for the
package maintainers.

Cheers,

Matthew
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517: Build system API

[Matthew Brett]

Hi,

On Wed, Nov 23, 2016 at 6:35 AM, Thomas Kluyver  wrote:
> I'd like to push PEP 517 forwards again. This PEP specifies a general
> build system interface so that a source tree can specify a tool (such as
> flit), and pip can ask that tool to generate a wheel. This would allow
> us to install from an sdist or a VCS checkout without running a setup.py
> file.
>
> https://www.python.org/dev/peps/pep-0517/
>
> Developments since the last time this was discussed (to my knowledge):
> - It now uses the pyproject.toml file specified in PEP 518, which was
> already accepted. 517 originally specified a JSON file; 518 explains why
> TOML is preferred (basically: comments).
> - I have implemented the proposed build-system API in a pull request for
> Flit; this has been fairly straightforward so far.
> https://github.com/takluyver/flit/pull/77
>
> Questions:
> 1. Editable installs. The PEP currenly specifies a hook to do an
> editable install (like 'pip install -e' or 'setup.py develop') into a
> given prefix. I don't think that specifying a prefix is sufficient to
> cover '--user' installation, which uses a different install scheme,
> especially on Windows and OSX framework builds. We can:
> a. Add an extra parameter 'user' to the hook, to override the prefix and
> do a user install.
> b. Leave it as is, and do not support editable user installation (which
> would make me sad, as I do editable user installs regularly)
> c. Decide that editable installs are too fiddly to standardise, and
> leave it to users to invoke a tool directly to do an editable install.

I think the standard advice nowadays is to do user installs via pip,
so I would vote for some mechanism to do this - option a).

> 2. Dash vs. underscore, bikeshed reloaded! Currently,  the table name
> uses a dash: [build-system], but the key added by PEP 517 uses an
> underscore: build_backend. This seems a bit messy. I propose that we
> change build_backend to build-backend for consistency. Dashes and
> underscores can both be used in a TOML key without needing quoting.

dash seems good to me too,

Cheers,

Matthew
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Current Python packaging status (from my point of view)

[Matthew Brett]

Hi,

On Fri, Nov 4, 2016 at 11:36 PM, Nick Coghlan <ncogh...@gmail.com> wrote:
> On 4 November 2016 at 03:56, Matthew Brett <matthew.br...@gmail.com> wrote:
>> But - it would be a huge help if the PSF could help with funding to
>> get mingw-w64 working.  This is the crucial blocker for progress on
>> binary wheels on Windows.
>
> Such a grant was already awarded earlier this year by way of the
> Scientific Python Working Group (which is a collaborative funding
> initiative between the PSF and NumFocus):
> https://mail.python.org/pipermail/scientific/2016-January/000271.html
>
> However, we hadn't received a status update by the time I stepped down
> from the Board, although it sounds like progress hasn't been good if
> folks aren't even aware that the grant was awarded in the first place.

Ah no, that was a related project, but for a different set of work.
Specifically that was a 5K grant to Carl Kleffner to configure and
package up the mingw-w64 compilers to make it easier to build
extensions for Pythons prior to 3.5 - see [1].

Cheers,

Matthew

[1] https://mingwpy.github.io/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Current Python packaging status (from my point of view)

[Matthew Brett]

Hi,

On Thu, Nov 3, 2016 at 2:29 AM, Paul Moore <p.f.mo...@gmail.com> wrote:
> On 3 November 2016 at 00:02, Matthew Brett <matthew.br...@gmail.com> wrote:
>> Anaconda has an overwhelming advantage on Windows, in that Continuum
>> can bear the licensing liabilities enforced by the Intel Fortran
>> compiler, and we can not.  We therefore have no license-compatible
>> Fortran compiler for Python 3.5 - so we can't build scipy, and that's
>> a full stop for the scipy stack.   I'm sure you know, but the only
>> practical open-source option is mingw-w64, that does not work with the
>> Microsoft runtime used by Python 3.5 [1].  It appears the only person
>> capable of making mingw-w64 compatible with this runtime is Ray
>> Donnelly, who now works for Continuum, and we haven't yet succeeded in
>> finding the 15K USD or so to pay Continuum for his time.
>
> Is this something the PSF could assist with? Either in terms of
> funding work to get mingw-w64 working, or in terms of funding (or
> negotiating) Intel Fortran licenses for the community?

I'm afraid paying for the Intel Fortran licenses won't help because
the problem is in the Intel Fortran runtime license [1].

But - it would be a huge help if the PSF could help with funding to
get mingw-w64 working.  This is the crucial blocker for progress on
binary wheels on Windows.   Nathaniel - do you have any recent news on
progress?

Cheers,

Matthew

[1] https://software.intel.com/en-us/comment/1881526
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Current Python packaging status (from my point of view)

[Matthew Brett]

Hi,

On Wed, Nov 2, 2016 at 4:08 PM, Chris Barker  wrote:
> Hey Matthew,
>
>> > [1] There seems to be some animosity among pip supporters and conda
>> > supports, or at least a perception that there is.
>>
>> I don't know whether there is animosity, but there is certainly
>> tension.  Speaking personally, I care a lot about having the option to
>> prefer pip.
>
>
> indeed -- and you have made herculean efforts to make that possible :-)
>
>>
>> There are others in the scientific community who feel we
>> should standardize on conda.   I think this is the cause of Chris'
>> frustration.  If we could all use conda instead of pip, then this
>> would make it easier for him in terms of packaging, because he would
>> not have to support pip  (Chris please correct me if I'm wrong).
>
>
> yup -- nor would you :-)
>
> My personal frustration comes with my history -- I've been in this community
> for over 15 years -- and I spent a lot of effort back in the day to make
> packages available for the Mac (before pypi, pip, before wheels, ...). And I
> found I was constantly re-figuring out how to build the dependent libraries
> needed, and statically linking everything, etc... It sucked. A lot of work,
> and not at all fun (for me, anyway -- maybe some folks enjoy that sort of
> thing).
>
> So I started a project to share that effort, and to build a bit of
> infrastructure. I also started looking into how to handle dependent libs
> with pip+wheel -- I got no support whatsoever for that. I got frustrated
> 'cause it was too hard, and I also felt like I was fighting the tools. I did
> not get far.
>
> Mathew ended up picking up that effort and really making it work, and had
> gotten all the core SciPY stuff out there as binary wheels -- really great
> stuff.
>
> But then I discovered conda -- and while I was resistant at first, I found
> that it was a much nicer environment to do what I needed to do. It started
> not because of anything specific about conda, but because someone had
> already built a bunch of the stuff I needed -- nice!
>
> But the breaking point was when I needed to build a package of my own: py_gd
> -- it required libgd, which required libpng, libjpeg, libtiff --- some of
> those come out of the box on a Mac, it's all available from homebrew, and
> all pretty common on Linux -- but I have users that are on Windows, or on a
> Mac and can't grok homebrew or macports. And, frankly, neither do all of my
> development team.

Anaconda has an overwhelming advantage on Windows, in that Continuum
can bear the licensing liabilities enforced by the Intel Fortran
compiler, and we can not.  We therefore have no license-compatible
Fortran compiler for Python 3.5 - so we can't build scipy, and that's
a full stop for the scipy stack.   I'm sure you know, but the only
practical open-source option is mingw-w64, that does not work with the
Microsoft runtime used by Python 3.5 [1].  It appears the only person
capable of making mingw-w64 compatible with this runtime is Ray
Donnelly, who now works for Continuum, and we haven't yet succeeded in
finding the 15K USD or so to pay Continuum for his time.

> But not pyHDF, netCDF5, gdal, shapely, ... (to name a few that I need to
> work with). And these are ugly: which means very hard for end-users to
> build, and very hard for people to package up into wheels (is it even
> possible?)

I'd be surprised if these packages were very hard to build OSX and
Linux wheels for.  We're already building hard packages like Pillow
and Matplotlib and h5py and pytables.  If you mean netCDF4 - there are
already OSX and Windows wheels for that [2].

Cheers,

Matthew

[1] https://mingwpy.github.io/issues.html#the-vs-14-2015-runtime
[2] https://pypi.python.org/pypi/netCDF4
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Current Python packaging status (from my point of view)

[Matthew Brett]

Hi,

On Wed, Nov 2, 2016 at 11:31 AM, Donald Stufft  wrote:
[snip]
> [1] There seems to be some animosity among pip supporters and conda
> supports, or at least a perception that there is. I’d just like to say that
> this isn’t really shared (to my knowledge) by the development teams of
> either project. I think everyone involved thinks folks should use whatever
> solution best allows them to solve whatever problem they are having.

I don't know whether there is animosity, but there is certainly
tension.  Speaking personally, I care a lot about having the option to
prefer pip.  There are others in the scientific community who feel we
should standardize on conda.   I think this is the cause of Chris'
frustration.  If we could all use conda instead of pip, then this
would make it easier for him in terms of packaging, because he would
not have to support pip  (Chris please correct me if I'm wrong).

Although there are clear differences in the audience for pip and
conda, there is also a very large overlap.   In practice the majority
of users could reasonably choose one or the other as their starting
point.Of course, one may come to dominate this choice over the
other.  At the point where enough users become frustrated with the
lack of pip wheels, conda will become the default.   If pip wheels are
widely available, that makes the pressure to use conda less.  If we
reach this tipping point it will become wasteful of developer effort
to make pip wheels / conda packages, the number and quality of binary
packages will drop, and one of these package managers will go into
decline.

Cheers,

Matthew
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Current Python packaging status (from my point of view)

[Matthew Brett]

Hi,

On Tue, Nov 1, 2016 at 10:35 AM, Thomas Güttler
<guettl...@thomas-guettler.de> wrote:
>
>
> Am 01.11.2016 um 10:50 schrieb Nick Coghlan:
>>
>> On 1 November 2016 at 17:30, Thomas Güttler
>> <guettl...@thomas-guettler.de> wrote:
>>>
>>> Am 17.09.2016 um 12:29 schrieb Nick Coghlan:
>>>>
>>>> Hi folks,
>>>>
>>>> Prompted by a few posts I read recently about the current state of the
>>>> Python packaging ecosystem, I figured it made sense to put together an
>>>> article summarising my own perspective on the current state of things:
>>>>
>>>> http://www.curiousefficiency.org/posts/2016/09/python-packaging-ecosystem.html
>>>
>>>
>>>
>>> Thank you for this summarizing article. Yes, a lot was done during the
>>> last months.
>>>
>>> I liked the part "My core software ecosystem design philosophy" a lot,
>>> since
>>> it explains that both parties (software consumer and software publisher)
>>> want
>>> it to be simple and easy.
>>>
>>>
>>> About conda: if pip and conda overlap in some point. Why not implement
>>> this in a reusable library which gets used by conda and pip?
>>
>>
>> For the parts where they genuinely overlap, conda is already able to
>> just use pip, or else the same libraries that pip uses. For the
>> platform management pieces (SAT solving for conda repositories,
>> converting PyPI packages to conda ones, language independent
>> environment management), what conda does is outside the scope of what
>> pip supports anyway.
>>
>>> About funding: Looking for more income is one way to solve this. Why not
>>> look into the other direction: How to reduce costs?
>>
>>
>> Thanks to donated infrastructure, the direct costs to the PSF are
>> incredibly low already. Donald went into some detail on that in
>> https://caremad.io/posts/2016/05/powering-pypi/ and that's mostly
>> still accurate (although his funded role with HPE ended recently)
>>
>>> Heading "Making the presence of a compiler on end user systems optional".
>>> Here
>>> I just can say: Thank you very much. I guess it was a lot of hard work to
>>> make this all simple and easy for the software consumers and publishers.
>>> Thank you.
>>>
>>> I wrote some lines, but I deleted my thoughts about the topic "Automating
>>> wheel creation", since
>>> I am a afraid it could raise bad mood in this list again. That's not my
>>> goal.
>>
>>
>> I currently see 3 main ways that could eventually happen:
>>
>> - the PSF sorts out its general sustainability concerns to the point
>> where it believes it can credibly maintain such a service on the
>> community's behalf
>> - the conda-forge folks branch out into offering wheel building as
>> well (so it becomes a matter of "publish your Python projects for the
>> user level conda platform, get platform independent Python wheels as
>> well")
>> - someone builds such a service independently of the current PyPI
>> infrastructure team, and convinces package publishers to start using
>> it
>>
>> There's also a 4th variant, which is getting to a point where someone
>> figures out a pushbutton solution for a build pipeline in a public
>> PaaS that offers a decent free tier. This is potentially one of the
>> more promising options, since it means the sustainability risks
>> related to future growth in demand accrue to the PaaS providers,
>> rather than to the PSF. However, it's somewhat gated on the Warehouse
>> migration, since you really want API token support for that kind of
>> automation, which is something the current PyPI code base doesn't
>> have, and isn't going to gain.
>
>
> I like this 4th variant. I guess most companies which use Python
> in a professional way already run a own pypi server.
>
> I am unsure if a public PaaS for this would exist. Maybe a script
> which runs a container on linux is enough. At least enough to
> build linux-only wheels. I guess most people have root access
> to a linux server somewhere.

I wrote some scripts to do this:

https://github.com/matthew-brett/multibuild

See the README there for standard usage.   For simple projects the
project configuration is only a few lines.  Quite a few projects use
multibuild and travis-ci to build 32- and 64-bit manylinux and OSX
wheels, here are some (more complex) examples:

https://travis-ci.org/MacPython/numpy-wheels
https://travis-

Re: [Distutils] Module Installation Issues

[Matthew Brett]

Hi,

On Tue, Sep 13, 2016 at 3:55 PM, Donald Stufft  wrote:
>
>> On Sep 13, 2016, at 6:41 PM, Steve Dower  wrote:
>>
>> I think it's one of these things where we should suck it up and let the 90% 
>> case work fine, then display a big fat warning if anything weird may have 
>> happened and let users sort it out themselves.
>
>
> I am unsure. One of the really egregious and hard to debug weirdness is going 
> to be something like:
>
>
 import foo.bar  # foo, and foo.bar are in sys.modules
 pip.install(“thing”)  # This implicitly upgrades foo
 import foo.widget  # the old foo is in sys.modules, but the new foo.widget.
>
> The 90% case works when it’s *only* pure python and there’s no 
> upgrading/downgrading involved, however you can’t control whether there are 
> going to be upgrades/downgrades or not when dependencies are in play because 
> of ==, >=, >, <, <=, etc.
>
> Another problem, particularly on Windows, is going to be stuff like:
>
 import someclibrary
 pip.install(“thing”)  # Implicitly upgrades someclibrary
>
> In this case, this would explode because Python will have the 
> someclibrary.dll locked and the upgrade will attempt to remove it.
>
> My fears here are that people are going to get really confused when they make 
> state modifications to their Python environment and they don’t see those 
> modifications reflected (because of sys.modules caching or what have you). 
> You can possibly get around that with reload(), but we already know the 
> problems that reload() has. Of course, that doesn’t stop Python from offering 
> reload(), but I think the difference is that reload() is targeted towards 
> people with weird edge cases who can understand the caveats while the 
> hypothetical ``pip.install()`` is targeted towards people who are beginners 
> who are still having problems figuring out the difference between the REPL 
> and their shell, much less the vagaries of Python’s global state and 
> attempting to modify that in a now running Python program.
>
> Perhaps a better idea would be to add some smarts to the REPL (but not to 
> Python itself) that would detect something like:
>
 pip install
>
> And print a better error message that gives a better indication about what’s 
> gone wrong besides a SyntaxError?

I was thinking the same thing.  If it could also catch accidental

>>> python3 -m pip install

commands, that would be even better.

Cheers,

Matthew
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Request for comment: Proposal to change behaviour of pip install

[Matthew Brett]

On Sun, Jun 26, 2016 at 11:59 AM, Matthew Brett <matthew.br...@gmail.com> wrote:
> Hi,
>
> On Sun, Jun 26, 2016 at 10:32 AM, Donald Stufft <don...@stufft.io> wrote:
>>
>> On Jun 25, 2016, at 6:25 AM, Pradyun Gedam <pradyu...@gmail.com> wrote:
>>
>> There is currently a proposal to change the behaviour to pip install to
>> upgrade a package that is passed even if it is already installed.
>>
>> This behaviour change is accompanied with a change in the upgrade strategy -
>> pip would stop “eagerly” upgrading dependencies and would become more
>> conservative, upgrading a dependency only when it doesn’t meet lower
>> constraints of the newer version of a parent. Moreover, the behaviour of pip
>> install --target would also be changed so that --upgrade no longer affects
>> it.
>>
>>
>> I think bundling these two changes (and I think I might have been the one
>> that originally suggested it) is making this discussion harder than it needs
>> to be as folks are having to fight on multiple different fronts at once. I
>> think the change to the default behavior of pip install is dependent on the
>> change to —upgrade, so I suggest we focus on the change to —upgrade first,
>> changing from a “recursive” to a “conservative” strategy. Once we get that
>> change figured out and landed then we can worry about what to do with pip
>> install.
>>
>> I’m not going to repeat the entire post, but I just made a fairly lengthy
>> comment at https://github.com/pypa/pip/issues/3786#issuecomment-228611906
>> but to try and boil it down to a few points:
>>
>> * ``pip install —upgrade`` is not a good security mechanism, relying on it
>> is inconsistent at best. If we want to support trying to keep people on
>> secure versions of software we need a better mechanism than this anyways, so
>> we shouldn’t let it influence our choice here.
>> * For the general case, it’s not going to matter a lot which way we go, but
>> not upgrading has the greatest chance of not breaking *already installed
>> software*.
>> * For the hard-to-upgrade case, the current behavior is so bad that people
>> are outright attempting to subvert the way pip typically behaviors, *AND*
>> advocating for other’s to do the same, in an attempt to escape that
>> behavior. I think that this is not a good place to be in.
>
> I wonder whether it is worth going back to the proposal [1] to add
>
> pip upgrade
>
> To anyone who hasn't read [1], this would have the behavior proposed
> (always upgrades named packages, does not do recursive upgrade).
> Meanwhile `pip install` stays as is, but deprecates the `--upgrade`
> flag in favor of the new command.
>
> The cost of the new command, that duplicates some behavior of
> `install` - seems rather small - and we could always deprecate it
> later, once people had got used the new behavior.

And for the specific case of `pip install pkg` always upgrading, that
seems like a bad idea.

There are two sensible things that `pip install pkg` could do, one is
checking and not upgrading (current behavior), the other is upgrading
(proposed change).

I think it would piss a lot of people off if we change from one
sensible behavior to another without a significant degree of prior
warning.

Cheers,

Matthew
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Request for comment: Proposal to change behaviour of pip install

[Matthew Brett]

Hi,

On Sun, Jun 26, 2016 at 10:32 AM, Donald Stufft  wrote:
>
> On Jun 25, 2016, at 6:25 AM, Pradyun Gedam  wrote:
>
> There is currently a proposal to change the behaviour to pip install to
> upgrade a package that is passed even if it is already installed.
>
> This behaviour change is accompanied with a change in the upgrade strategy -
> pip would stop “eagerly” upgrading dependencies and would become more
> conservative, upgrading a dependency only when it doesn’t meet lower
> constraints of the newer version of a parent. Moreover, the behaviour of pip
> install --target would also be changed so that --upgrade no longer affects
> it.
>
>
> I think bundling these two changes (and I think I might have been the one
> that originally suggested it) is making this discussion harder than it needs
> to be as folks are having to fight on multiple different fronts at once. I
> think the change to the default behavior of pip install is dependent on the
> change to —upgrade, so I suggest we focus on the change to —upgrade first,
> changing from a “recursive” to a “conservative” strategy. Once we get that
> change figured out and landed then we can worry about what to do with pip
> install.
>
> I’m not going to repeat the entire post, but I just made a fairly lengthy
> comment at https://github.com/pypa/pip/issues/3786#issuecomment-228611906
> but to try and boil it down to a few points:
>
> * ``pip install —upgrade`` is not a good security mechanism, relying on it
> is inconsistent at best. If we want to support trying to keep people on
> secure versions of software we need a better mechanism than this anyways, so
> we shouldn’t let it influence our choice here.
> * For the general case, it’s not going to matter a lot which way we go, but
> not upgrading has the greatest chance of not breaking *already installed
> software*.
> * For the hard-to-upgrade case, the current behavior is so bad that people
> are outright attempting to subvert the way pip typically behaviors, *AND*
> advocating for other’s to do the same, in an attempt to escape that
> behavior. I think that this is not a good place to be in.

I wonder whether it is worth going back to the proposal [1] to add

pip upgrade

To anyone who hasn't read [1], this would have the behavior proposed
(always upgrades named packages, does not do recursive upgrade).
Meanwhile `pip install` stays as is, but deprecates the `--upgrade`
flag in favor of the new command.

The cost of the new command, that duplicates some behavior of
`install` - seems rather small - and we could always deprecate it
later, once people had got used the new behavior.

Cheers,

Matthew

[1] 
https://gist.github.com/pradyunsg/4c9db6a212239fee69b429c96cdc3d73#add-a-pip-upgrade-command
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] If you want wheel to be successful, provide a build server.

[Matthew Brett]

Hi,

On Thu, May 26, 2016 at 11:47 AM, Donald Stufft <don...@stufft.io> wrote:
>
>> On May 26, 2016, at 2:41 PM, Matthew Brett <matthew.br...@gmail.com> wrote:
>>
>> On Thu, May 26, 2016 at 2:28 PM, Daniel Holth <dho...@gmail.com> wrote:
>>> Maybe there could be a way to say "the most recent release that has a wheel
>>> for my platform". That would help with the problem of binaries not being
>>> available concurrently with a new source distribution.
>>
>> Yes, that would certainly help get over some of the immediate problems.
>>
>> Sorry for my ignorance - but does ``--only-binary`` search for an
>> earlier release with a binary or just bomb out if the latest release
>> does not have a binary?   It would also be good to have a flag to say
>> "if this is pure Python go ahead and use the source, otherwise error".
>>   Otherwise I guess we'd have to rely on everyone with a pure Python
>> package generating wheels.
>
> I believe it would find the latest version that has a wheel available,
> I could be misremembering though.

Reflecting a bit more on this - how easy would be be to add a flag
``--prefer-binary`` that would have the effect of:

* Installing a binary wheel for current release if available, otherwise;
* Checking previous release for binary wheel, installing if present, otherwise;
* Install from sdist

I think that would help a great deal in reducing surprise in wheel
installs while we get better at putting up wheels at the same time as
sdists.

Cheers,

Matthew
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] If you want wheel to be successful, provide a build server.

[Matthew Brett]

On Thu, May 26, 2016 at 2:28 PM, Daniel Holth  wrote:
> Maybe there could be a way to say "the most recent release that has a wheel
> for my platform". That would help with the problem of binaries not being
> available concurrently with a new source distribution.

Yes, that would certainly help get over some of the immediate problems.

Sorry for my ignorance - but does ``--only-binary`` search for an
earlier release with a binary or just bomb out if the latest release
does not have a binary?   It would also be good to have a flag to say
"if this is pure Python go ahead and use the source, otherwise error".
   Otherwise I guess we'd have to rely on everyone with a pure Python
package generating wheels.

It would be very good to work out a plan for new Python releases as
well.  We really need to get wheels up to pypi a fair while before the
release date, and it's easy to forget to do that, because, at the
moment, we don't have much testing infrastructure to make sure that a
range of wheel installs are working OK.

Cheers,

Matthew
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] If you want wheel to be successful, provide a build server.

[Matthew Brett]

Hi,

On Thu, May 26, 2016 at 1:57 AM, Nick Coghlan  wrote:
> On 26 May 2016 at 01:11, Thomas Güttler  wrote:
>>
>>
>> Am 25.05.2016 um 15:52 schrieb Nick Coghlan:
>>>
>>> On 25 May 2016 at 17:13, Thomas Güttler 
>>> wrote:

 If you want wheel to be successful, **provide a build server**.
>>>
>>>
>>> Thomas, aside from that statement being demonstrably untrue (since the
>>> wheel format has already proven to be wildly successful, even with
>>> developers coping with Linux ABI fragmentation), an attitude of "give
>>> me more free stuff, or your project will fail" is not an acceptable
>>> tone to adopt on this list.
>>>
>>> The contributors here are (mainly) volunteers working on
>>> infrastructure provided by a public interest charity, not your
>>> personal servants to be ordered about as you feel inclined.
>>
>> You seem to be angry. Why?
>
> Thomas, I am pointing out that your current exhibition of entitled
> behaviour across multiple venues (as represented by the specific
> sentence I quoted) is problematic. Please stop trying to crack the
> whip and generate an artificial sense of urgency - software
> publication and distribution is a complex problem, and most of the
> current challenges in the PyPI ecosystem stem from an ongoing lack of
> funding which requires organisational change moreso than technical
> change. While various folks are working on that, it's mostly not a
> distutils-sig level problem, but rather a question for the PSF and for
> commercial Python redistributors.
>
> If you're looking to provide information, or ask if a particular
> solution that seems obvious to you would be feasible in practice, then
> do that. The one thing I am asking you to STOP doing is combining your
> questions with exaggerated hyperbole that's designed to make
> volunteers feel bad.

I just wanted to make sure that we didn't lose out on starting a
discussion of this problem.

The problem is of course caused by the runaway success of the wheel
format, and I'm sure it can be solved in a sensible way, but however
expressed, it's true that wheels have become so standard that we do
need to think about automation for build and release, if we aren't
going run into trouble.   By trouble, I mean that users will often hit
the situation where they don't get wheels when they expect to, and get
turned off pypi / wheels as a result.  I have personally put a great
deal of work into building and releasing wheels, so that is something
I'd really like to avoid.

So - can I humbly ask - what is the best way to get that discussion going?

Cheers,

Matthew
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] If you want wheel to be successful, provide a build server.

[Matthew Brett]

On Wed, May 25, 2016 at 11:22 AM, Thomas Güttler
<guettl...@thomas-guettler.de> wrote:
>
>
> Am 25.05.2016 um 15:55 schrieb Paul Moore:
>>
>> On 25 May 2016 at 14:42, Thomas Güttler <guettl...@thomas-guettler.de>
>> wrote:
>>>
>>> Am 25.05.2016 um 09:57 schrieb Alex Grönholm:
>>>>
>>>>
>>>> Amen to that, but who will pay for it? I imagine a great deal of
>>>> processing power would be required for this.
>>>> How do implementors of other languages handle this?
>>>
>>>
>>>
>>> I talked with someone who is member of the python software foundation,
>>> and
>>> he said that
>>> money for projects like this is available. Of course this was no official
>>> statement.
>>
>>
>> The other aspect of this is who has sufficient time/expertise to set
>> something like this up? Are you volunteering to do this?
>
>
> I am volunteering for doing coordination work:
>  - communication
>  - layout of datastructures
>  - interchange of datastructures.
>  - no coding
>
> But we need at least ten people how say "I'm willing to help"

We (scientific Python folks) have been thinking about this too [1].

We're getting to the stage where the informal methods we have been
using are difficult to coordinate.  As more people provide wheels, it
gets more common for packages to release source before wheels, and
cause a cry of pain from users whose installation suddenly fails or
changes.

The situation got particularly bad when Python 3.5 was released,
because none or very few of us had Python 3.5 packages ready, and so
nearly all new installs were suddenly not getting wheels, and
suffering.

We all really need some system that can, from some simple trigger,
like a push of a git tag, build wheels for Windows 32 + 64, OSX, and
manylinux1_x86_64, for Pythons 2.7, 3.4 and 3.5 (and 3.6?), test these
installs, and, if the tests pass, and push these either to an
accessible spot (we have been using donated rackspace hosting) or to
pypi directly.

We're fairly close to that at the moment.  Many of us are building and
testing binaries on Appveyor, Travis by default.   We have lots of
projects set up with repos MacPython github org, where the
repositories only exist to build, test OSX wheels on travis-ci OSX
VMs, and push the wheels to rackspace - e.g. [2].   We have been
setting up similar systems for manylinux - e.g. [3, 4].

The problem is that these systems are largely manual, in that a
release for e.g. numpy involves:

* Trigger build / test on separate Appveyor repo;
* Trigger build / test on MacPython/numpy-wheels;
* Trigger build on manylinux-builds / test on manylinux testing;
* When all these are done and tests passing, locate generated binaries
on rackspace / Appveyor, and upload to pypi.

This is complicated, and it's relatively hard for a given package to
set this up for themselves.

When Python 3.6 comes out, we'll all have to do this release procedure
at more or less the same time.

So, I would love to have a system that could either collate these
different services (Appveyor, Travis, Circle-CI, Rackspace, AWS) into
something coherent, or generate something new that is more
streamlined.  See conda-forge [5] for an example of collating build
services.

I think it's fine for each package to specify its own build and test
recipes as long as they can do it in a way that is well defined, with
examples to work from.  The success of travis-ci is a testament to the
ingenuity of packagers in getting their packages built and tested.

Maybe this could be a PEP of its own.  It would certainly help to have
some idea of what kind of support the PSF can give - the spec would
look different for a new custom system and a system collating Appveyor
/ Travis etc.

I'm certainly happy to devote time to this (in the hope of saving a
lot of time later).

Best,

Matthew

[1] https://github.com/scipy/scipy/issues/6157#issuecomment-219314029
[2] https://github.com/MacPython/numpy-wheels
[3] https://github.com/matthew-brett/manylinux-builds
[4] https://github.com/matthew-brett/manylinux-testing
[5] https://conda-forge.github.io/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Does anyone understand what's going on with libpython on Linux?

[Matthew Brett]

On Sun, Feb 7, 2016 at 4:38 AM, Antoine Pitrou  wrote:
> On Sun, 7 Feb 2016 00:25:57 -0800
> "Robert T. McGibbon"  wrote:
>> > What are Debian/Ubuntu doing in distutils so that extensions don't link
>> to libpython by default?
>>
>> I don't know exactly, but one way to reproduce this is simply to build the
>> interpreter without `--enable-shared`.
>
> See https://bugs.python.org/issue21536. It would be nice if you could
> lobby for this issue to be resolved... (though that would only be for
> 3.6, presumably)

Just to unpack from that issue - and quoting a nice summary by you (Antoine):

"... the -l flag was added in #832799, for a rather complicated case
where the interpreter is linked with a library dlopened by an
embedding application (I suppose for some kind of plugin system)."

Following the link to https://bugs.python.org/issue832799 - the `-l`
flag (and therefore the dependency on libpython was added at Python
2.3 for the case where an executable A dlopens a library B.so . B.so
has an embedded Python interpreter and is linked to libpython.
However, when the embedded Python interpreter in B.so loads an
extension module mymodule.so , mymodule.so does not inherit a
namespace with the libpython symbols already loaded. See
https://bugs.python.org/msg18810 .

One option we have then is to remove all DT_NEEDED references to
libpython in manylinux wheels. We get instant compatibility for bare
Debian / Ubuntu Python installs, at the cost of causing some puzzling
crash for the case of: dlopened library with embedded Python
interpreter where the embedded Python interpreter imports a manylinux
wheel.

On the other hand, presumably this same crash will occur for nearly
all Debian-packaged Python extension modules (if it is true that they
do not specify a libpython dependency) - so it seems unlikely that
this is a common problem.

Cheers,

Matthew
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Does anyone understand what's going on with libpython on Linux?

[Matthew Brett]

On Sun, Feb 7, 2016 at 2:19 PM, Robert T. McGibbon  wrote:
>> One option we have then is to remove all DT_NEEDED references to
> libpython in manylinux wheels. We get instant compatibility for bare
> Debian / Ubuntu Python installs, at the cost of causing some puzzling
> crash for the case of: dlopened library with embedded Python
> interpreter where the embedded Python interpreter imports a manylinux
> wheel.
>
> I don't think this is acceptable, since it's going to break some packages
> that depend
> on dlopen.
>
>> On the other hand, presumably this same crash will occur for nearly
> all Debian-packaged Python extension modules (if it is true that they
> do not specify a libpython dependency) - so it seems unlikely that
> this is a common problem.
>
> I don't think so. Debian-packaged extensions that require libpython to exist
> (a minority of them to be sure, but ones that use complex shared library
> layouts)
> just declare a dependency on libpython. For example, python-pyside has a
> Depends on libpython2.7:
>
> ```
> $ apt-cache depends python-pyside.qtcore
> python-pyside.qtcore
>   Depends: libc6
>   Depends: libgcc1
>   Depends: libpyside1.2
>   Depends: libpython2.7
>   Depends: libqtcore4
>   Depends: libshiboken1.2v5
>   Depends: libstdc++6
>   Depends: python
>   Depends: python
>   Conflicts: python-pyside.qtcore:i386
> ```

Sure - and this might be because the pyside packager was being
especially careful about libpython, or it might be an accident -
pyside is hard to build.

On the other hand, it looks like almost all the common Debian packages
don't declare this dependency - so almost all of the standard
scientific Python stack and more would crash in this corner case:

apt-cache depends python-numpy | grep libpython
apt-cache depends python-scipy | grep libpython
apt-cache depends python-yaml | grep libpython
apt-cache depends python-regex | grep libpython
apt-cache depends python-matplotlib | grep libpython

It seems reasonable to build to the same compatibility level as most
Debian packaged modules.

Matthew
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Matthew Brett]

On Sun, Jan 31, 2016 at 11:25 PM, Marius Gedminas <mar...@gedmin.as> wrote:
> On Sat, Jan 30, 2016 at 12:17:02PM -0800, Matthew Brett wrote:
>> Hi,
>>
>> > I can confirm that Debian and Anaconda builds of CPython 2.7 both have
>> > sys.maxunicode == 0x10, but Enthought Canopy has sys.maxunicode ==
>> > 0x. Hmm. I guess they should fix that.
>> >
>> > Also the manylinux docker image currently has sys.maxunicode ==
>> > 0x, so we should definitely fix that :-).
>>
>> A quick check on Ubuntu 12.04, Debian sid, Centos 7.2 confirms wide
>> unicode by default.  Are there any known distributions providing UCS2
>> unicode Pythons?
>
> I don't know of any.

I also tested on Debian wheezy 32-bit; Fedora 22 (32-bit packages);
openSUSE 13.2, recent Arch and Gentoo, these are all wide unicode.

> Pyenv, OTOH, deliberately uses upstream defaults and so produces narrow
> unicode builds.

Ouch - good to know.

Cheers,

Matthew
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions

[Matthew Brett]

Hi,

> I can confirm that Debian and Anaconda builds of CPython 2.7 both have
> sys.maxunicode == 0x10, but Enthought Canopy has sys.maxunicode ==
> 0x. Hmm. I guess they should fix that.
>
> Also the manylinux docker image currently has sys.maxunicode ==
> 0x, so we should definitely fix that :-).

A quick check on Ubuntu 12.04, Debian sid, Centos 7.2 confirms wide
unicode by default.  Are there any known distributions providing UCS2
unicode Pythons?

Cheers,

Matthew
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] draft PEP: manylinux1

[Matthew Brett]

On Thu, Jan 21, 2016 at 11:05 AM, Matthew Brett <matthew.br...@gmail.com> wrote:
> Hi,
>
> On Thu, Jan 21, 2016 at 2:05 AM, M.-A. Lemburg <m...@egenix.com> wrote:
>> On 21.01.2016 10:31, Nick Coghlan wrote:
>>> On 21 January 2016 at 19:03, M.-A. Lemburg <m...@egenix.com> wrote:
>>>> By using the version based approach, we'd not run into this
>>>> problem and gain a lot more.
>>>
>>> I think it's better to start with a small core that we *know* works,
>>> then expand later, rather than trying to make the first iteration too
>>> wide. The "manylinux1" tag itself is versioned (hence the "1" at the
>>> end), so "manylinux2" may simply have *more* libraries defined, rather
>>> than newer ones.
>>
>> My argument is that the file based approach taken by the PEP
>> is too limiting to actually make things work for a large
>> set of Python packages.
>>
>> It will basically only work for packages that do not interface
>> to other external libraries (except for the few cases listed in
>> the PEP, e.g. X11, GL, which aren't always installed or
>> available either).
>>
>> IMO, testing the versions of a set of libraries is a safer
>> approach. It's perfectly fine to have a few dependencies
>> not work in a module because an optional system package is not
>> installed, e.g. say a package comes with UIs written in
>> Qt and one in GTK.
>
> Please forgive my slowness, but I don't understand exactly what you
> mean.  Can you give a specific example?
>
> Say my package depends on libpng.
>
> Call the machine I'm installing on the client machine.
>
> Are you saying that, when I build a wheel, I should specify to the
> wheel what versions of libpng I can tolerate on the the client
> machine, and if if the client does have a compatible version, then pip
> should raise an error, perhaps with a useful message about how to get
> libpng?

Sorry, slowness any typos - corrected:

Are you saying that, when I build a wheel, I should specify to the
wheel what versions of libpng I can tolerate on the the client
machine, and if the client does _not_ have a compatible version, then pip
should raise an error, perhaps with a useful message about how to get
libpng?

Best again,

Matthew
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] draft PEP: manylinux1

[Matthew Brett]

Hi,

On Thu, Jan 21, 2016 at 11:57 AM, Chris Barker <chris.bar...@noaa.gov> wrote:
> On Thu, Jan 21, 2016 at 11:48 AM, Matthew Brett <matthew.br...@gmail.com>
> wrote:
>>
>> > So without a plan to provide all that stuff -- I"m not sure of the
>> > utility
>> > of this -- how are you gong to get PIL/Pillow to work? statically link
>> > up
>> > the ying-yang? Not sure the linux world will take to that.
>>
>> That's exactly how the current OSX Pillow wheels work, and they've
>> been working fine for a while now.  There just aren't that many
>> libraries to worry about for the vast majority of packages.
>
>
> well, that was really intended to be only an example. And OS-X provides more
> basic libs than manylinux anyway (zlib, freetype, either png or jpeg, can't
> remember which).
>
> The library list got long enough to drive me crazy -- I guess you've got
> more patience than I have. Tried to build any OSGEO stuff lately?

I'm sure there are packages that would be hard to build, but for the
vast majority of packages, getting a build recipe is a one-time only
job which might (in bad cases) take a day or so, and then can be
maintained from time to time by the package maintainer.   The Pillow
wheel builder is a good example - I built the prototype, but the
Pillow guys own and maintain it now.

I don't think it's sensible to veto all linux wheels because there are
some packages that will be hard to build.

Cheers,

Matthew
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] draft PEP: manylinux1

[Matthew Brett]

Hi,

On Thu, Jan 21, 2016 at 11:42 AM, Chris Barker  wrote:
> nice, idea, but
>
> libX11.so.6
> libXext.so.6
> libXrender.so.1
> libGL.so.1
>
> These are all X11, yes? pretty much any workstation will have these, but in
> general, servers won't.
>
> Someone on this thread suggested that that's OK -- don't expect a GUI
> package to work on a linux box without a GUI. But some of these libs might
> get used for stuff like back-end rendering, etc, so would be expected to
> work on a headless box. I think Anaconda an Canopy have gotten away with
> this because both of their user bases are primarily desktop data analysis
> type of stuff -- not web services, web servers, etc.

Someone administering a headless server will surely be able to cope
with that problem, and the trade-off of convenience for the large
majority of users seems like a good one to me.

> Then there is the "additional libs" problem. Again, Anaconda and Canopy can
> do this because they are providing those libs - lots of stuff you'd
> generally expect to be there in a typical *nix system stuff like libpng,
> libjpeg, who knows what they heck else.
>
> So without a plan to provide all that stuff -- I"m not sure of the utility
> of this -- how are you gong to get PIL/Pillow to work? statically link up
> the ying-yang? Not sure the linux world will take to that.

That's exactly how the current OSX Pillow wheels work, and they've
been working fine for a while now.  There just aren't that many
libraries to worry about for the vast majority of packages.

Cheers,

Matthew
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] draft PEP: manylinux1

[Matthew Brett]

Hi,

On Thu, Jan 21, 2016 at 7:45 PM, Robert T. McGibbon  wrote:
> On Thu, Jan 21, 2016 at 7:32 PM, Nick Coghlan  wrote:
>>
>>
>> However, it does suggest a possible alternative approach to naming
>> these compatibility subsets: what if the name of this particular
>> platform compatibility tag was something like "linux-sciabi1", rather
>> than "manylinux1"?
>
>
> That's an interesting idea, but I personally don't see the manylinux1 list
> as particularly
> "scientific". If anything, I'd call it "minimal".

Yes, I agree, I don't think 'linux-sciabi1" would differentiate this
from other ways of building wheels.  For example, I can't see why this
wouldn't be a perfectly reasonable way to proceed for someone doing
audio or video.   The difference that "manylinux" was designed to
capture is the idea of having a single set of wheels for many versions
of Linux, rather than wheels specific to particular distributions or
packaged versions of external libraries.

Cheers,

Matthew
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPi not allowing duplicate filenames

[Matthew Brett]

Hi,

On Wed, Oct 14, 2015 at 11:25 AM, Nathaniel Smith  wrote:
> On Oct 14, 2015 11:12 AM, "Donald Stufft"  wrote:
>>
> [...]
>>> Apparently some packages were making assumptions about the format of the
>>> numpy.__version__ string, and having .postN in there caused errors when they
>>> tried to process it. (It would be helpful if there were a little
>>> permissively licensed standalone implementation of PEP 440 comparisons,
>>> suitable for the "if pkg.version > ...:" checks that people insist on doing
>>> -- I couldn't find one in some quick searches.)
>>
>> https://github.com/pypa/packaging
>>
>> It’s what both pip and setuptools use (though we embed it, but it’s fine
>> to depend on it too).
>
> That's under Apache 2, so it can't be used by GPLv2 packages, or any package
> that might be used by GPLv2 packages.
>
>>>
>>> IIUC, the specific problems numpy ran into that caused the creation of
>>> .postN releases were:
>>> - oops, didn't sign the uploads, re-upload identical file with proper
>>> signature attached -> not allowed. (I'm not sure if these were embedded or
>>> detached signatures. Either way it'd be nice if pypi allowed it, but for
>>> embedded signatures in particular I can see how this might be a hassle.)
>>
>> I don’t think we allow embedded signatures, it would be reasonable to
>> allow uploading detached signatures after the fact though.
>>>
>>>
>>> - our OS X maintainer tried to use twine to upload OS X wheels for the
>>> existing release; instead it created a new release. Not sure if a bug was
>>> filed on twine, but if not then one probably should be. As a workaround our
>>> release docs now say "always upload wheels by hand using the web interface,
>>> never use setup.py upload or twine".
>>
>> This shouldn’t create a new release unless you’ve changed the version
>> number (including adding post releases). If you can reproduce on Test PyPI I
>> can fix it.
>
> Matthew? Any thoughts?

Sorry - I'm not sure whether y'all are surprised by the behavior of
twine in this situation.

Specifically, the situation was:

* There was a release on pypi was labeled 1.10.0 :
https://pypi.python.org/pypi/numpy/1.10.0;
* The source archives that Chuck uploaded were from code that gave a
version string of '1.10.0.post2';
* I built wheels from this code, that had this version encoded in the
filename - e.g.
http://wheels.scikit-image.org/numpy-1.10.0.post2-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
;
* I used `twine upload
http://wheels.scikit-image.org/numpy-1.10.0.post2-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl`
etc to upload these.

When I did this, pypi created a new release called
https://pypi.python.org/pypi/numpy/1.10.0.post2 and put the wheels
there.

I'm guessing that this is what y'all expected?

Best,

Matthew
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] pip on windows: life without eggs ok?

[Matthew Brett]

Hi,

On Sun, Apr 15, 2012 at 8:07 PM, Chris Lambacher ch...@kateandchris.net wrote:
 Hi,

 If I were telling a new Windows Python user what to do to get started, I
 would tell them to use easy_install because easy_install will pick up both
 installers and binary eggs and do the right thing with those and
 virtualenvs.

 pip on Windows is a decent into misery.

Seconded; our scientific packages use numpy (C) and scipy (C, C++
Fortran, big) and have C and sometimes Fortran extensions, so pip is
more or less useless for windows, and a real PITA for anything else.

Best,

Matthew
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [buildout] branches languishing? (site-packages and distutils scripts)

[Matthew Brett]

Hi,

On Sun, Mar 25, 2012 at 2:49 PM, Hanno Schlichting ha...@hannosch.eu wrote:
 On Sun, Mar 25, 2012 at 11:31 PM, Jim Fulton j...@zope.com wrote:
 Hanno suggested creating a buildout organization.  I'm up with that.
 Maybe someone can create one and include me. :)

 In case you want it, there you go: https://github.com/buildout/buildout

 Jim is a co-owner of the organization having admin access, the
 developers team with pull/push rights is currently empty.

I wrote a little sphinx project that is simple to drop in to your
sphinx docs to seed a github workflow:

https://github.com/matthew-brett/gitwash

http://matthew-brett.github.com/pydagogue/gitwash_build.html

Best,

Matthew
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Specify missing install-requires dependencies to develop egg?

[Matthew Brett]

Hi,

On Fri, Aug 26, 2011 at 6:05 AM, Jim Fulton j...@zope.com wrote:
 On Thu, Aug 25, 2011 at 9:06 PM, Matthew Brett matthew.br...@gmail.com 
 wrote:
 Hi,

 We have been working on using buildout with a develop package
 depending on numpy, and ran into trouble because 'numpy' is not
 specified in the package 'install_requires'.

 Specifically, we have something like this:

 [buildout]
 parts =
    nibabel-target
    mypy

 [nibabel-target]
 recipe = zc.recipe.egg:develop
 setup = src/nibabel

 [mypy]
 recipe = zc.recipe.egg:script
 eggs =
    nibabel
 interpreter = py

 'nibabel' needs to import numpy to install, but does not specify
 'numpy' in its setup.py install_requires, because pip install of numpy
 is fragile, and we wanted to error early in general for an attempted
 pip install of nibabel, if numpy was not present.

 But of course the script above raises an error for the
 'nibabel-target' part, because of missing numpy:

 -
 RuntimeError: Cannot import package numpy - is it installed?
 While:
  Installing nibabel-target.

 An internal error occurred due to a bug in either zc.buildout or in a
 recipe being used:
 

 What we would very much like to do, is to be able to specify the
 missing install_requires for this package (and for others we are
 thinking of working with), in the recipe - something like:

 [nibabel-target]
 recipe = zc.recipe.egg:develop
 setup = src/nibabel
 install-requires =
    numpy

 but we couldn't immediately see a way of specifying such a dependency.
  A quick look at the Develop code in zc/recipe/egg/custom.py
 suggested that there wasn't an easy way to add some needed egg onto
 the path at this point.

 Is there anything we missed that we should have tried?

 If not, what would be the best way to solve our problem?  A modified recipe?

 Theoretically, you'd use setup_requires to specify packages needed by setup.
 I'm not sure if that works with buildout. I've heard reports that it
 does and reports that
 it doesn't.

Ah - yes - thanks - I had forgotten about setup_requires.

 Why does nibabel want numpy at setup time? If it's to get C incluse files, 
 then
 you may be out of luck, as it's unlikely that buildout will install
 numpy in a way that
 will give nibabel what it wants.

As I was saying in my previous message, nibabel just needs numpy at
runtime.  However, various of our other packages need it at setup
time, to find C include files, or to use numpy.distutils.

When running these various failing buildouts, numpy got installed as
an egg into the ./egg directory.  If I put that egg onto the path with
PYTHONPATH in my bare virtualenv, and then rerun this basic original
buildout.cfg

[buildout]
parts =
nibabel-target
mypy

[nibabel-target]
recipe = zc.recipe.egg:develop
setup = src/nibabel

[mypy]
recipe = zc.recipe.egg:script
eggs =
nibabel
interpreter = py

then it runs to completion.  With ./bin/py:

 import numpy
 numpy.get_include()
'/home/mb312/tmp/buildbotting/eggs/numpy-1.6.1-py2.6-linux-x86_64.egg/numpy/core/include'

- so I think the C includes would work.

Looking at the bugs pointed to by Wichert, it sounds like your
previous suggestion of a 'setup-requires' for custom and develop
recipes might be the way to go - that was what I was thinking of but
mis-naming in my previous mail.

Cheers,

Matthew
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] Specify missing install-requires dependencies to develop egg?

[Matthew Brett]

Hi,

We have been working on using buildout with a develop package
depending on numpy, and ran into trouble because 'numpy' is not
specified in the package 'install_requires'.

Specifically, we have something like this:

[buildout]
parts =
nibabel-target
mypy

[nibabel-target]
recipe = zc.recipe.egg:develop
setup = src/nibabel

[mypy]
recipe = zc.recipe.egg:script
eggs =
nibabel
interpreter = py

'nibabel' needs to import numpy to install, but does not specify
'numpy' in its setup.py install_requires, because pip install of numpy
is fragile, and we wanted to error early in general for an attempted
pip install of nibabel, if numpy was not present.

But of course the script above raises an error for the
'nibabel-target' part, because of missing numpy:

-
RuntimeError: Cannot import package numpy - is it installed?
While:
  Installing nibabel-target.

An internal error occurred due to a bug in either zc.buildout or in a
recipe being used:


What we would very much like to do, is to be able to specify the
missing install_requires for this package (and for others we are
thinking of working with), in the recipe - something like:

[nibabel-target]
recipe = zc.recipe.egg:develop
setup = src/nibabel
install-requires =
numpy

but we couldn't immediately see a way of specifying such a dependency.
  A quick look at the Develop code in zc/recipe/egg/custom.py
suggested that there wasn't an easy way to add some needed egg onto
the path at this point.

Is there anything we missed that we should have tried?

If not, what would be the best way to solve our problem?  A modified recipe?

Best,

Matthew
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Specify missing install-requires dependencies to develop egg?

[Matthew Brett]

Hi,

On Thu, Aug 25, 2011 at 7:06 PM, Reinout van Rees rein...@vanrees.org wrote:
 On 26-08-11 03:06, Matthew Brett wrote:

 [mypy]
 recipe = zc.recipe.egg:script
 eggs =
     nibabel
 interpreter = py

 ...

 What we would very much like to do, is to be able to specify the
 missing install_requires for this package (and for others we are
 thinking of working with)

 The solution is quite simple: just add numpy to the list of eggs like this:

 [mypy]
 recipe = zc.recipe.egg:script
 eggs =
    nibabel
    numpy
 interpreter = py

That doesn't work for me. Rightly or wrongly I am guessing that the
problem is that the dependency is not for the interpreter, but for
running nibabel setup.py.  In any case the failure remains with that
addition:

Updating nibabel-target.
Traceback (most recent call last):
  File /tmp/tmpnKAmK6, line 11, in module
execfile('/home/mb312/tmp/buildbotting/src/nibabel/setup.py')
  File /home/mb312/tmp/buildbotting/src/nibabel/setup.py, line 40, in module
package_check('numpy', NUMPY_MIN_VERSION)
  File /home/mb312/tmp/buildbotting/src/nibabel/nisext/sexts.py,
line 122, in package_check
raise RuntimeError(msgs['missing'] % pkg_name)
RuntimeError: Cannot import package numpy - is it installed?
While:
  Updating nibabel-target.

An internal error occurred due to a bug in either zc.buildout or in a
recipe being used:
...

Best,

Matthew
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Specify missing install-requires dependencies to develop egg?

[Matthew Brett]

Hi,

On Thu, Aug 25, 2011 at 7:50 PM, Reinout van Rees rein...@vanrees.org wrote:
 On 26-08-11 03:06, Matthew Brett wrote:

 'nibabel' needs to import numpy to install

 With your latest reply I re-read your original. Hm, this looks like a real
 problem: if your setup.py already imports from numpy (which your statement
 sounds like), numpy really needs to be there already.

Yes, that is what made me think that a new recipe might be the only
way, where the recipe would put numpy on the path during the develop
step.

 The only thing I can think of right now is to have numpy installed globally
 and to use buildout 1.4.4 + osc.recipe.sysegg to depend on numpy. If you
 call the sysegg part right at the beginning, you might get it to work.

Yes, if numpy is already on the path, it works, but we wanted to test
against several different numpy versions, and it seemed ugly and
fragile to make a separate virtualenv for each python / numpy version
pair.  We'd much rather use the buildout mechanism to specify these,
if possible.

 Alternative is to re-work your setup.py. Do you really need to import numpy?

Actually, no, not in this case, it is only used as a crude dependency
check.  But, for other packages we have, we do need numpy.distutils
for the setup.py - for example to find the numpy include directory for
compiling C modules.  The only reason I was offering this package as
an example is that it is rather small and easy to install compared to
the others.

Thanks a lot,

Matthew
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig