Re: [Distutils] If you want wheel to be successful, provide a build server.

[Pradyun Gedam]

(this is my first mail to the list, hopefully, this goes through)

Hey Matthew,

FYI, the --prefer-binary flag that you mention has come up in earlier
discussions and already has an issue over at pip’s github repo (
https://github.com/pypa/pip/issues/3785).

Regards,
Pradyun
​

On Tue, 14 Jun 2016 at 00:21 Matthew Brett  wrote:

> Hi,
>
> On Thu, May 26, 2016 at 11:47 AM, Donald Stufft  wrote:
> >
> >> On May 26, 2016, at 2:41 PM, Matthew Brett 
> wrote:
> >>
> >> On Thu, May 26, 2016 at 2:28 PM, Daniel Holth  wrote:
> >>> Maybe there could be a way to say "the most recent release that has a
> wheel
> >>> for my platform". That would help with the problem of binaries not
> being
> >>> available concurrently with a new source distribution.
> >>
> >> Yes, that would certainly help get over some of the immediate problems.
> >>
> >> Sorry for my ignorance - but does ``--only-binary`` search for an
> >> earlier release with a binary or just bomb out if the latest release
> >> does not have a binary?   It would also be good to have a flag to say
> >> "if this is pure Python go ahead and use the source, otherwise error".
> >>   Otherwise I guess we'd have to rely on everyone with a pure Python
> >> package generating wheels.
> >
> > I believe it would find the latest version that has a wheel available,
> > I could be misremembering though.
>
> Reflecting a bit more on this - how easy would be be to add a flag
> ``--prefer-binary`` that would have the effect of:
>
> * Installing a binary wheel for current release if available, otherwise;
> * Checking previous release for binary wheel, installing if present,
> otherwise;
> * Install from sdist
>
> I think that would help a great deal in reducing surprise in wheel
> installs while we get better at putting up wheels at the same time as
> sdists.
>
> Cheers,
>
> Matthew
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] If you want wheel to be successful, provide a build server.

[Matthew Brett]

Hi,

On Thu, May 26, 2016 at 11:47 AM, Donald Stufft  wrote:
>
>> On May 26, 2016, at 2:41 PM, Matthew Brett  wrote:
>>
>> On Thu, May 26, 2016 at 2:28 PM, Daniel Holth  wrote:
>>> Maybe there could be a way to say "the most recent release that has a wheel
>>> for my platform". That would help with the problem of binaries not being
>>> available concurrently with a new source distribution.
>>
>> Yes, that would certainly help get over some of the immediate problems.
>>
>> Sorry for my ignorance - but does ``--only-binary`` search for an
>> earlier release with a binary or just bomb out if the latest release
>> does not have a binary?   It would also be good to have a flag to say
>> "if this is pure Python go ahead and use the source, otherwise error".
>>   Otherwise I guess we'd have to rely on everyone with a pure Python
>> package generating wheels.
>
> I believe it would find the latest version that has a wheel available,
> I could be misremembering though.

Reflecting a bit more on this - how easy would be be to add a flag
``--prefer-binary`` that would have the effect of:

* Installing a binary wheel for current release if available, otherwise;
* Checking previous release for binary wheel, installing if present, otherwise;
* Install from sdist

I think that would help a great deal in reducing surprise in wheel
installs while we get better at putting up wheels at the same time as
sdists.

Cheers,

Matthew
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] If you want wheel to be successful, provide a build server.

[Robin Becker]

As a lurker on this list and long time user of everything python I feel this is 
becoming appropriate


https://xkcd.com/927/
--
Robin Becker
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] If you want wheel to be successful, provide a build server.

[Chris Barker]

On Thu, May 26, 2016 at 11:22 PM, Wes Turner  wrote:

>
> "conda-forge  is a github organization
> containing repositories of conda recipes.
>

Just to be clear -- while conda-forge is about conda packages, a good deal
of the CI integration stuff would apply just as well to building wheels.

It would make a lot of sense to build off of what they've done.

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] If you want wheel to be successful, provide a build server.

[Wes Turner]

conda-forge/conda-smithy

| Src: https://github.com/conda-forge/conda-smithy
| Homepage: https://conda-forge.github.io/ :

"conda-forge  is a github organization
containing repositories of conda recipes. Thanks to some awesome continuous
integration providers (AppVeyor, CircleCI and TravisCI), each repository,
also known as a feedstock, automatically builds its own recipe in a clean
and repeatable way on Windows, Linux and OSX"

https://conda-forge.github.io/feedstocks.html

so, for psutil-feedstock:

https://github.com/conda-forge/psutil-feedstock

-
https://github.com/conda-forge/psutil-feedstock/blob/master/recipe/meta.yaml
- https://github.com/conda-forge/psutil-feedstock/blob/master/appveyor.yml
-
https://github.com/conda-forge/psutil-feedstock/blob/master/ci_support/run_docker_build.sh
- https://github.com/conda-forge/psutil-feedstock/blob/master/.travis.yml

…
https://github.com/conda-forge/conda-smithy/tree/master/conda_smithy/templates

On Thursday, May 26, 2016, Robert T. McGibbon  wrote:

> > I want to get something setup that would allow people to only need to
> upload
> > a source release to PyPI and then have wheels automatically built for
> them
> > (but not mandate that- Projects that wish it should always be able to
> control
> > their wheel generation). I don’t know what that would specifically look
> > like, if someone is motivated to work on it I’m happy to help figure out
> what
> > it should look like and provide guidance where I can, otherwise it’ll
> wait
> > until I get around to it.
>
> One first step towards this that's a natural follow-on to the manylinux
> work
> might be to define a overall build configuration file / format and process
> for
> automating the whole wheel build cycle (i'm thinking of something modeled
> after
> conda-build) that would, among other things
>
> for potentially multiple versions of python:
> - run `pip wheel` (or setu.py bdist_wheel) to compile the wheel
> - run `auditwheel` (linux) or `delocate` (osx) to bundle any external
> libraries
>
> -Robert
>
> On Thu, May 26, 2016 at 2:47 PM, Donald Stufft  > wrote:
>
>>
>> > On May 26, 2016, at 2:41 PM, Matthew Brett > > wrote:
>> >
>> > On Thu, May 26, 2016 at 2:28 PM, Daniel Holth > > wrote:
>> >> Maybe there could be a way to say "the most recent release that has a
>> wheel
>> >> for my platform". That would help with the problem of binaries not
>> being
>> >> available concurrently with a new source distribution.
>> >
>> > Yes, that would certainly help get over some of the immediate problems.
>> >
>> > Sorry for my ignorance - but does ``--only-binary`` search for an
>> > earlier release with a binary or just bomb out if the latest release
>> > does not have a binary?   It would also be good to have a flag to say
>> > "if this is pure Python go ahead and use the source, otherwise error".
>> >   Otherwise I guess we'd have to rely on everyone with a pure Python
>> > package generating wheels.
>>
>> I believe it would find the latest version that has a wheel available,
>> I could be misremembering though.
>>
>> >
>> > It would be very good to work out a plan for new Python releases as
>> > well.  We really need to get wheels up to pypi a fair while before the
>> > release date, and it's easy to forget to do that, because, at the
>> > moment, we don't have much testing infrastructure to make sure that a
>> > range of wheel installs are working OK.
>> >
>>
>> I want to get something setup that would allow people to only need to
>> upload
>> a source release to PyPI and then have wheels automatically built for them
>> (but not mandate that- Projects that wish it should always be able to
>> control
>> their wheel generation). I don’t know what that would specifically look
>> like, if someone is motivated to work on it I’m happy to help figure out
>> what
>> it should look like and provide guidance where I can, otherwise it’ll wait
>> until I get around to it.
>>
>> —
>> Donald Stufft
>>
>>
>>
>> ___
>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>> 
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>
>
>
> --
> -Robert
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] If you want wheel to be successful, provide a build server.

[Nick Coghlan]

On 27 May 2016 10:40, "Nick Coghlan"  wrote:
>
>
> On 27 May 2016 04:48, "Donald Stufft"  wrote:
> > > On May 26, 2016, at 2:41 PM, Matthew Brett 
wrote:
> > > It would be very good to work out a plan for new Python releases as
> > > well.  We really need to get wheels up to pypi a fair while before the
> > > release date, and it's easy to forget to do that, because, at the
> > > moment, we don't have much testing infrastructure to make sure that a
> > > range of wheel installs are working OK.
> >
> > I want to get something setup that would allow people to only need to
upload
> > a source release to PyPI and then have wheels automatically built for
them
> > (but not mandate that- Projects that wish it should always be able to
control
> > their wheel generation).
>
> A possible preceding step for that might be to create a service that
reports per-project information on clients downloading the sdist versions
of a project. With the Big Query data publicly available, that shouldn't
need to be part of PyPI itself (at least in the near term), so it should
just need an interested volunteer, rather than being gated on Warehouse.

It belatedly occurs to me that experimenting with that myself would align
pretty well with some ideas I'm exploring for $(day job), so if nobody else
pursues this, I'll likely take a look myself some time in the next few
weeks (and if they do take a look, I should be able to find the time to
help).

Cheers,
Nick.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] If you want wheel to be successful, provide a build server.

[Nick Coghlan]

On 27 May 2016 04:48, "Donald Stufft"  wrote:
> > On May 26, 2016, at 2:41 PM, Matthew Brett 
wrote:
> > It would be very good to work out a plan for new Python releases as
> > well.  We really need to get wheels up to pypi a fair while before the
> > release date, and it's easy to forget to do that, because, at the
> > moment, we don't have much testing infrastructure to make sure that a
> > range of wheel installs are working OK.
>
> I want to get something setup that would allow people to only need to
upload
> a source release to PyPI and then have wheels automatically built for them
> (but not mandate that- Projects that wish it should always be able to
control
> their wheel generation).

A possible preceding step for that might be to create a service that
reports per-project information on clients downloading the sdist versions
of a project. With the Big Query data publicly available, that shouldn't
need to be part of PyPI itself (at least in the near term), so it should
just need an interested volunteer, rather than being gated on Warehouse.

The intent there would be to allow projects that decide to build their own
wheels to prioritise their wheel creation, and be able to quantify the
impact of each addition to their build matrix, rather than necessarily
providing pre-built binaries for all supported platforms supported by the
source release.

The other thing that data could be used for is to start quantifying the
throughput requirements for an "all of PyPI" build service by looking at
release publication rates (rather than download rates). Again, likely
pursuable by volunteers using the free tier of a public PaaS, rather than
requiring ongoing investment.

Cheers,
Nick.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] If you want wheel to be successful, provide a build server.

[Paul Moore]

On 26 May 2016 at 19:20, Matthew Brett  wrote:
> I just wanted to make sure that we didn't lose out on starting a
> discussion of this problem.
>
> The problem is of course caused by the runaway success of the wheel
> format, and I'm sure it can be solved in a sensible way, but however
> expressed, it's true that wheels have become so standard that we do
> need to think about automation for build and release, if we aren't
> going run into trouble.   By trouble, I mean that users will often hit
> the situation where they don't get wheels when they expect to, and get
> turned off pypi / wheels as a result.  I have personally put a great
> deal of work into building and releasing wheels, so that is something
> I'd really like to avoid.
>
> So - can I humbly ask - what is the best way to get that discussion going?

I would certainly like to promote projects using existing CI systems
for building wheels. It seems like a pretty simple approach that would
cover a decent proportion of the problem.

There's already a section in the Packaging User Guide explaining how
to use Appveyor to build Windows binary wheels. If you don't have
dependencies to deal with (e.g., you're simply providing a C speedup
module) it's pretty simple. If you *do* have dependencies to deal
with, I suspect you could arrange for them to be uploaded when you
build somehow without too much issue - but I don't have projects like
that so I can't really add that to the document. The biggest problem
with Appveyor is that it's less well known than Travis. However, I
don't know of many (frankly, any!) projects using this approach -
whether that's because of a lack of awareness, or because it's not as
useful as it seemed to me when I wrote it, or some other reason, I
don't honestly know. Maybe we could get some feedback on what can be
done to improve that section? Is there going to be any sort of
packaging discussions at PyCon? Maybe it's a question that could be
asked? (I'm not going to be there myself, unfortunately).

For Linux/Unix, the problem was always the plethora of binary formats.
If we can assume that manylinux builds will constitute a 90% solution
here, maybe someone could contribute a section explaining how to add
the building of manylinux wheels into a project's Travis config. I
know almost nothing about OSX, but doesn't Travis offer OSX builders?
Could they be used to build extensions in the same way as the above?

Going beyond this level (which is basically making sure projects know
they have easy access to build environments for the key platforms, and
encouraging them to use them) is when it gets harder. Integrating
artifacts from travis/appveyor into a release process would need some
work, for example. Maybe a way forward here would be for some projects
to try doing so, and writing up a set of "how we did it" notes that
could over time turn into a set of recommended practices in the PUG.
Or an interested 3rd party could probably do most of the
experimentation and development, simply by forking a few typical
projects and seeing what was needed.

With all of the above, we'd end up with some (hopefully!) pretty good
documentation of how to (relatively) painlessly add building of wheels
for all major platforms into *any* project (that didn't have
specialised issues to deal with).

I'm not sure where we'd go from there. The next step seems to be to
actually provide some sort of build service, or curated set of wheels
for projects that don't/won't/can't produce their own. But that's a
lot of work - the conda folks, and people like Christoph Gohlke, know
how much. Maybe by making the process simple enough (as described
above) might encourage someone to put together a wheel service along
the lines of Christoph's (but for all platforms, and in a format that
pip can access directly). Once something like that starts up, it may
attract additional volunteers, and grow from there. But it's still
quite a commitment from someone.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] If you want wheel to be successful, provide a build server.

[Robert T. McGibbon]

> I want to get something setup that would allow people to only need to
upload
> a source release to PyPI and then have wheels automatically built for them
> (but not mandate that- Projects that wish it should always be able to
control
> their wheel generation). I don’t know what that would specifically look
> like, if someone is motivated to work on it I’m happy to help figure out
what
> it should look like and provide guidance where I can, otherwise it’ll wait
> until I get around to it.

One first step towards this that's a natural follow-on to the manylinux work
might be to define a overall build configuration file / format and process
for
automating the whole wheel build cycle (i'm thinking of something modeled
after
conda-build) that would, among other things

for potentially multiple versions of python:
- run `pip wheel` (or setu.py bdist_wheel) to compile the wheel
- run `auditwheel` (linux) or `delocate` (osx) to bundle any external
libraries

-Robert

On Thu, May 26, 2016 at 2:47 PM, Donald Stufft  wrote:

>
> > On May 26, 2016, at 2:41 PM, Matthew Brett 
> wrote:
> >
> > On Thu, May 26, 2016 at 2:28 PM, Daniel Holth  wrote:
> >> Maybe there could be a way to say "the most recent release that has a
> wheel
> >> for my platform". That would help with the problem of binaries not being
> >> available concurrently with a new source distribution.
> >
> > Yes, that would certainly help get over some of the immediate problems.
> >
> > Sorry for my ignorance - but does ``--only-binary`` search for an
> > earlier release with a binary or just bomb out if the latest release
> > does not have a binary?   It would also be good to have a flag to say
> > "if this is pure Python go ahead and use the source, otherwise error".
> >   Otherwise I guess we'd have to rely on everyone with a pure Python
> > package generating wheels.
>
> I believe it would find the latest version that has a wheel available,
> I could be misremembering though.
>
> >
> > It would be very good to work out a plan for new Python releases as
> > well.  We really need to get wheels up to pypi a fair while before the
> > release date, and it's easy to forget to do that, because, at the
> > moment, we don't have much testing infrastructure to make sure that a
> > range of wheel installs are working OK.
> >
>
> I want to get something setup that would allow people to only need to
> upload
> a source release to PyPI and then have wheels automatically built for them
> (but not mandate that- Projects that wish it should always be able to
> control
> their wheel generation). I don’t know what that would specifically look
> like, if someone is motivated to work on it I’m happy to help figure out
> what
> it should look like and provide guidance where I can, otherwise it’ll wait
> until I get around to it.
>
> —
> Donald Stufft
>
>
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>



-- 
-Robert
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] If you want wheel to be successful, provide a build server.

[Donald Stufft]

> On May 26, 2016, at 2:41 PM, Matthew Brett  wrote:
> 
> On Thu, May 26, 2016 at 2:28 PM, Daniel Holth  wrote:
>> Maybe there could be a way to say "the most recent release that has a wheel
>> for my platform". That would help with the problem of binaries not being
>> available concurrently with a new source distribution.
> 
> Yes, that would certainly help get over some of the immediate problems.
> 
> Sorry for my ignorance - but does ``--only-binary`` search for an
> earlier release with a binary or just bomb out if the latest release
> does not have a binary?   It would also be good to have a flag to say
> "if this is pure Python go ahead and use the source, otherwise error".
>   Otherwise I guess we'd have to rely on everyone with a pure Python
> package generating wheels.

I believe it would find the latest version that has a wheel available,
I could be misremembering though.

> 
> It would be very good to work out a plan for new Python releases as
> well.  We really need to get wheels up to pypi a fair while before the
> release date, and it's easy to forget to do that, because, at the
> moment, we don't have much testing infrastructure to make sure that a
> range of wheel installs are working OK.
> 

I want to get something setup that would allow people to only need to upload
a source release to PyPI and then have wheels automatically built for them
(but not mandate that- Projects that wish it should always be able to control
their wheel generation). I don’t know what that would specifically look
like, if someone is motivated to work on it I’m happy to help figure out what
it should look like and provide guidance where I can, otherwise it’ll wait
until I get around to it.

—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] If you want wheel to be successful, provide a build server.

[Matthew Brett]

On Thu, May 26, 2016 at 2:28 PM, Daniel Holth  wrote:
> Maybe there could be a way to say "the most recent release that has a wheel
> for my platform". That would help with the problem of binaries not being
> available concurrently with a new source distribution.

Yes, that would certainly help get over some of the immediate problems.

Sorry for my ignorance - but does ``--only-binary`` search for an
earlier release with a binary or just bomb out if the latest release
does not have a binary?   It would also be good to have a flag to say
"if this is pure Python go ahead and use the source, otherwise error".
   Otherwise I guess we'd have to rely on everyone with a pure Python
package generating wheels.

It would be very good to work out a plan for new Python releases as
well.  We really need to get wheels up to pypi a fair while before the
release date, and it's easy to forget to do that, because, at the
moment, we don't have much testing infrastructure to make sure that a
range of wheel installs are working OK.

Cheers,

Matthew
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] If you want wheel to be successful, provide a build server.

[Donald Stufft]

> On May 26, 2016, at 2:36 PM, Donald Stufft  wrote:
> 
> 
>> On May 26, 2016, at 2:28 PM, Daniel Holth > > wrote:
>> 
>> Maybe there could be a way to say "the most recent release that has a wheel 
>> for my platform". That would help with the problem of binaries not being 
>> available concurrently with a new source distribution.
> 
> 
> 
> pip install —binary-only :all: ...
> 

Sorry, it’s —only-binary.


—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] If you want wheel to be successful, provide a build server.

[Donald Stufft]

> On May 26, 2016, at 2:28 PM, Daniel Holth  wrote:
> 
> Maybe there could be a way to say "the most recent release that has a wheel 
> for my platform". That would help with the problem of binaries not being 
> available concurrently with a new source distribution.



pip install —binary-only :all: ...

—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] If you want wheel to be successful, provide a build server.

[Daniel Holth]

Maybe there could be a way to say "the most recent release that has a wheel
for my platform". That would help with the problem of binaries not being
available concurrently with a new source distribution.

On Thu, May 26, 2016 at 2:21 PM Matthew Brett 
wrote:

> Hi,
>
> On Thu, May 26, 2016 at 1:57 AM, Nick Coghlan  wrote:
> > On 26 May 2016 at 01:11, Thomas Güttler 
> wrote:
> >>
> >>
> >> Am 25.05.2016 um 15:52 schrieb Nick Coghlan:
> >>>
> >>> On 25 May 2016 at 17:13, Thomas Güttler 
> >>> wrote:
> 
>  If you want wheel to be successful, **provide a build server**.
> >>>
> >>>
> >>> Thomas, aside from that statement being demonstrably untrue (since the
> >>> wheel format has already proven to be wildly successful, even with
> >>> developers coping with Linux ABI fragmentation), an attitude of "give
> >>> me more free stuff, or your project will fail" is not an acceptable
> >>> tone to adopt on this list.
> >>>
> >>> The contributors here are (mainly) volunteers working on
> >>> infrastructure provided by a public interest charity, not your
> >>> personal servants to be ordered about as you feel inclined.
> >>
> >> You seem to be angry. Why?
> >
> > Thomas, I am pointing out that your current exhibition of entitled
> > behaviour across multiple venues (as represented by the specific
> > sentence I quoted) is problematic. Please stop trying to crack the
> > whip and generate an artificial sense of urgency - software
> > publication and distribution is a complex problem, and most of the
> > current challenges in the PyPI ecosystem stem from an ongoing lack of
> > funding which requires organisational change moreso than technical
> > change. While various folks are working on that, it's mostly not a
> > distutils-sig level problem, but rather a question for the PSF and for
> > commercial Python redistributors.
> >
> > If you're looking to provide information, or ask if a particular
> > solution that seems obvious to you would be feasible in practice, then
> > do that. The one thing I am asking you to STOP doing is combining your
> > questions with exaggerated hyperbole that's designed to make
> > volunteers feel bad.
>
> I just wanted to make sure that we didn't lose out on starting a
> discussion of this problem.
>
> The problem is of course caused by the runaway success of the wheel
> format, and I'm sure it can be solved in a sensible way, but however
> expressed, it's true that wheels have become so standard that we do
> need to think about automation for build and release, if we aren't
> going run into trouble.   By trouble, I mean that users will often hit
> the situation where they don't get wheels when they expect to, and get
> turned off pypi / wheels as a result.  I have personally put a great
> deal of work into building and releasing wheels, so that is something
> I'd really like to avoid.
>
> So - can I humbly ask - what is the best way to get that discussion going?
>
> Cheers,
>
> Matthew
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] If you want wheel to be successful, provide a build server.

[Matthew Brett]

Hi,

On Thu, May 26, 2016 at 1:57 AM, Nick Coghlan  wrote:
> On 26 May 2016 at 01:11, Thomas Güttler  wrote:
>>
>>
>> Am 25.05.2016 um 15:52 schrieb Nick Coghlan:
>>>
>>> On 25 May 2016 at 17:13, Thomas Güttler 
>>> wrote:

 If you want wheel to be successful, **provide a build server**.
>>>
>>>
>>> Thomas, aside from that statement being demonstrably untrue (since the
>>> wheel format has already proven to be wildly successful, even with
>>> developers coping with Linux ABI fragmentation), an attitude of "give
>>> me more free stuff, or your project will fail" is not an acceptable
>>> tone to adopt on this list.
>>>
>>> The contributors here are (mainly) volunteers working on
>>> infrastructure provided by a public interest charity, not your
>>> personal servants to be ordered about as you feel inclined.
>>
>> You seem to be angry. Why?
>
> Thomas, I am pointing out that your current exhibition of entitled
> behaviour across multiple venues (as represented by the specific
> sentence I quoted) is problematic. Please stop trying to crack the
> whip and generate an artificial sense of urgency - software
> publication and distribution is a complex problem, and most of the
> current challenges in the PyPI ecosystem stem from an ongoing lack of
> funding which requires organisational change moreso than technical
> change. While various folks are working on that, it's mostly not a
> distutils-sig level problem, but rather a question for the PSF and for
> commercial Python redistributors.
>
> If you're looking to provide information, or ask if a particular
> solution that seems obvious to you would be feasible in practice, then
> do that. The one thing I am asking you to STOP doing is combining your
> questions with exaggerated hyperbole that's designed to make
> volunteers feel bad.

I just wanted to make sure that we didn't lose out on starting a
discussion of this problem.

The problem is of course caused by the runaway success of the wheel
format, and I'm sure it can be solved in a sensible way, but however
expressed, it's true that wheels have become so standard that we do
need to think about automation for build and release, if we aren't
going run into trouble.   By trouble, I mean that users will often hit
the situation where they don't get wheels when they expect to, and get
turned off pypi / wheels as a result.  I have personally put a great
deal of work into building and releasing wheels, so that is something
I'd really like to avoid.

So - can I humbly ask - what is the best way to get that discussion going?

Cheers,

Matthew
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] If you want wheel to be successful, provide a build server.

[Nick Coghlan]

On 26 May 2016 at 01:11, Thomas Güttler  wrote:
>
>
> Am 25.05.2016 um 15:52 schrieb Nick Coghlan:
>>
>> On 25 May 2016 at 17:13, Thomas Güttler 
>> wrote:
>>>
>>> If you want wheel to be successful, **provide a build server**.
>>
>>
>> Thomas, aside from that statement being demonstrably untrue (since the
>> wheel format has already proven to be wildly successful, even with
>> developers coping with Linux ABI fragmentation), an attitude of "give
>> me more free stuff, or your project will fail" is not an acceptable
>> tone to adopt on this list.
>>
>> The contributors here are (mainly) volunteers working on
>> infrastructure provided by a public interest charity, not your
>> personal servants to be ordered about as you feel inclined.
>
> You seem to be angry. Why?

Thomas, I am pointing out that your current exhibition of entitled
behaviour across multiple venues (as represented by the specific
sentence I quoted) is problematic. Please stop trying to crack the
whip and generate an artificial sense of urgency - software
publication and distribution is a complex problem, and most of the
current challenges in the PyPI ecosystem stem from an ongoing lack of
funding which requires organisational change moreso than technical
change. While various folks are working on that, it's mostly not a
distutils-sig level problem, but rather a question for the PSF and for
commercial Python redistributors.

If you're looking to provide information, or ask if a particular
solution that seems obvious to you would be feasible in practice, then
do that. The one thing I am asking you to STOP doing is combining your
questions with exaggerated hyperbole that's designed to make
volunteers feel bad.

Regards,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] If you want wheel to be successful, provide a build server.

[Noah Kantrowitz]

> On May 25, 2016, at 8:22 AM, Thomas Güttler  
> wrote:
> 
> 
> 
> Am 25.05.2016 um 15:55 schrieb Paul Moore:
>> On 25 May 2016 at 14:42, Thomas Güttler  wrote:
>>> Am 25.05.2016 um 09:57 schrieb Alex Grönholm:
 
 Amen to that, but who will pay for it? I imagine a great deal of
 processing power would be required for this.
 How do implementors of other languages handle this?
>>> 
>>> 
>>> I talked with someone who is member of the python software foundation, and
>>> he said that
>>> money for projects like this is available. Of course this was no official
>>> statement.
>> 
>> The other aspect of this is who has sufficient time/expertise to set
>> something like this up? Are you volunteering to do this?
> 
> I am volunteering for doing coordination work:
> - communication
> - layout of datastructures
> - interchange of datastructures.
> - no coding
> 
> But we need at least ten people how say "I'm willing to help"

Short answer: this is not how anything works.

Long answer: This is not a question of getting some number of people to help. 
If you can clone us a small army of Donalds, Nicks, and Richards then we 
_might_ be able to pull this off. The money isn't the problem per se, it's the 
human cost in upkeep for a system designed explicitly to run hostile code 
safely.

--Noah



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] If you want wheel to be successful, provide a build server.

[Paul Moore]

On 25 May 2016 at 19:03, Randy Syring  wrote:
> On 05/25/2016 11:48 AM, Paul Moore wrote:
>
>> I don't know about Nick, but your tone is certainly annoying to me. I
>> do this work in my (very limited!) spare time, and in most cases, what
>> I do is of no benefit to me personally, but is purely to help others
>> who raise issues.
>
> I wonder if this is an issue of English being Thomas' second language.  By
> his email address, he appears to reside in Germany.  So maybe something has
> been lost in translation?

I've certainly been happy to assume that it is, for the most part.
However, I do believe that there comes a point where even if there are
language issues, it's important to let people know how their comments
are coming across. I hope Thomas took my post in that vein.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] If you want wheel to be successful, provide a build server.

[Randy Syring]

On 05/25/2016 11:48 AM, Paul Moore wrote:

I don't know about Nick, but your tone is certainly annoying to me. I
do this work in my (very limited!) spare time, and in most cases, what
I do is of no benefit to me personally, but is purely to help others
who raise issues.


I wonder if this is an issue of English being Thomas' second language.  
By his email address, he appears to reside in Germany. So maybe 
something has been lost in translation?


*Randy Syring*
Husband | Father | Redeemed Sinner

/"For what does it profit a man to gain the whole world
and forfeit his soul?" (Mark 8:36 ESV)/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] If you want wheel to be successful, provide a build server.

[Sylvain Corlay]

For the build, one could use continuous integration providers like
appveyor, circleci and travisci to perform the builds. This is what has
been done by conda folks with conda-forge.



On Wed, May 25, 2016 at 11:51 AM, Chris Barker 
wrote:

> On Wed, May 25, 2016 at 8:27 AM, Thomas Güttler <
> guettl...@thomas-guettler.de> wrote:
>
>> I think providing a self hostable build server which can be started with
>> one command
>> would be such a project.
>
>
> The manylinux Docker container is heading in that direction already. I
> suggest you consider helping out with that effort.
>
> There is also the conda ecosystem, with the conda-forge project using
> public CI systems to build packages:
>
> https://conda-forge.github.io/
>
> I imagine a similar system could be adopted for wheels
>
> -CHB
>
>
>
>
>
>
>
>>
>> Regards,
>>   Thomas Güttler
>>
>>
>> --
>> Thomas Guettler http://www.thomas-guettler.de/
>> ___
>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
>>
>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R(206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115   (206) 526-6317   main reception
>
> chris.bar...@noaa.gov
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] If you want wheel to be successful, provide a build server.

[Chris Barker]

On Wed, May 25, 2016 at 8:27 AM, Thomas Güttler <
guettl...@thomas-guettler.de> wrote:

> I think providing a self hostable build server which can be started with
> one command
> would be such a project.


The manylinux Docker container is heading in that direction already. I
suggest you consider helping out with that effort.

There is also the conda ecosystem, with the conda-forge project using
public CI systems to build packages:

https://conda-forge.github.io/

I imagine a similar system could be adopted for wheels

-CHB







>
> Regards,
>   Thomas Güttler
>
>
> --
> Thomas Guettler http://www.thomas-guettler.de/
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] If you want wheel to be successful, provide a build server.

[Matthew Brett]

On Wed, May 25, 2016 at 11:22 AM, Thomas Güttler
 wrote:
>
>
> Am 25.05.2016 um 15:55 schrieb Paul Moore:
>>
>> On 25 May 2016 at 14:42, Thomas Güttler 
>> wrote:
>>>
>>> Am 25.05.2016 um 09:57 schrieb Alex Grönholm:


 Amen to that, but who will pay for it? I imagine a great deal of
 processing power would be required for this.
 How do implementors of other languages handle this?
>>>
>>>
>>>
>>> I talked with someone who is member of the python software foundation,
>>> and
>>> he said that
>>> money for projects like this is available. Of course this was no official
>>> statement.
>>
>>
>> The other aspect of this is who has sufficient time/expertise to set
>> something like this up? Are you volunteering to do this?
>
>
> I am volunteering for doing coordination work:
>  - communication
>  - layout of datastructures
>  - interchange of datastructures.
>  - no coding
>
> But we need at least ten people how say "I'm willing to help"

We (scientific Python folks) have been thinking about this too [1].

We're getting to the stage where the informal methods we have been
using are difficult to coordinate.  As more people provide wheels, it
gets more common for packages to release source before wheels, and
cause a cry of pain from users whose installation suddenly fails or
changes.

The situation got particularly bad when Python 3.5 was released,
because none or very few of us had Python 3.5 packages ready, and so
nearly all new installs were suddenly not getting wheels, and
suffering.

We all really need some system that can, from some simple trigger,
like a push of a git tag, build wheels for Windows 32 + 64, OSX, and
manylinux1_x86_64, for Pythons 2.7, 3.4 and 3.5 (and 3.6?), test these
installs, and, if the tests pass, and push these either to an
accessible spot (we have been using donated rackspace hosting) or to
pypi directly.

We're fairly close to that at the moment.  Many of us are building and
testing binaries on Appveyor, Travis by default.   We have lots of
projects set up with repos MacPython github org, where the
repositories only exist to build, test OSX wheels on travis-ci OSX
VMs, and push the wheels to rackspace - e.g. [2].   We have been
setting up similar systems for manylinux - e.g. [3, 4].

The problem is that these systems are largely manual, in that a
release for e.g. numpy involves:

* Trigger build / test on separate Appveyor repo;
* Trigger build / test on MacPython/numpy-wheels;
* Trigger build on manylinux-builds / test on manylinux testing;
* When all these are done and tests passing, locate generated binaries
on rackspace / Appveyor, and upload to pypi.

This is complicated, and it's relatively hard for a given package to
set this up for themselves.

When Python 3.6 comes out, we'll all have to do this release procedure
at more or less the same time.

So, I would love to have a system that could either collate these
different services (Appveyor, Travis, Circle-CI, Rackspace, AWS) into
something coherent, or generate something new that is more
streamlined.  See conda-forge [5] for an example of collating build
services.

I think it's fine for each package to specify its own build and test
recipes as long as they can do it in a way that is well defined, with
examples to work from.  The success of travis-ci is a testament to the
ingenuity of packagers in getting their packages built and tested.

Maybe this could be a PEP of its own.  It would certainly help to have
some idea of what kind of support the PSF can give - the spec would
look different for a new custom system and a system collating Appveyor
/ Travis etc.

I'm certainly happy to devote time to this (in the hope of saving a
lot of time later).

Best,

Matthew

[1] https://github.com/scipy/scipy/issues/6157#issuecomment-219314029
[2] https://github.com/MacPython/numpy-wheels
[3] https://github.com/matthew-brett/manylinux-builds
[4] https://github.com/matthew-brett/manylinux-testing
[5] https://conda-forge.github.io/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] If you want wheel to be successful, provide a build server.

[Paul Moore]

On 25 May 2016 at 16:11, Thomas Güttler  wrote:
>> The contributors here are (mainly) volunteers working on
>> infrastructure provided by a public interest charity, not your
>> personal servants to be ordered about as you feel inclined.
>
> You seem to be angry. Why?

I don't know about Nick, but your tone is certainly annoying to me. I
do this work in my (very limited!) spare time, and in most cases, what
I do is of no benefit to me personally, but is purely to help others
who raise issues.

Your comment "If you want wheel to be successful, **provide a build
server**" is both patronising (do you think no-one here is aware that
a build server would be beneficial?), inaccurate (wheels *are*
successful - ask anyone who's got benefit from them, it's not hard to
find such people) and demanding (you suggest others need to do work,
but offer none of your own time to help).

Many of your postings here have had a similar tone - insisting that
things "have to be" done a certain way, or that there "must" be a
solution in the form you prefer. I've tended to assume that this has
simply been unintentional, and maybe getting exaggerated by the lack
of non-verbal cues available in email, but you need to consider your
words more carefully if you don't want to start seriously offending
people. And that's not behaviour that's acceptable on this list (or
indeed, any of the other Python lists).

> I am just bringing information from A to B (from github issue
> to this list).

Your information was phrased as a demand. If you don't think that's
the case, I encourage you to re-read your own post.

> I guess the author of psutil is just a volunteer like you are.

Yes, and (on the issue you linked to) he politely explained why he
didn't provide Linux binaries and offered some approaches you could
take. You seemed to take that as somehow implying that he was
dissatisfied and posted "information" here implying that. I hope you
didn't mean it that way, but in doing so, as far as I can tell you
misrepresented him.

> What is the problem? I think the problem is that
> creating wheels for psutil is difficult.

Hardly. The problem is that Unix systems are very diverse, and trying
to maintain compatible builds for all the variations is more work than
a single volunteer can manage. The people working on manylinux on this
mailing list have put a *lot* of work into trying to alleviate that
issue. Maybe the psutil author hasn't heard of this, or maybe he's not
had the time to investigate or take advantage of it. Or maybe he
doesn't feel (or indeed, he's sure - after all, psutil is a very
low-level package) that manylinux isn't the right answer for his
project.

But just because you personally weren't willing to set up the
prerequisites to build a copy of psutil for yourself, doesn't mean
that "creating wheels for psutil is difficult".

> I think we all agree on this.

No.

> If not, if you think it is easy, then please tell us.

Building and publishing wheels for every platform variation your user
base might be interested *is* difficult, simply because of the scale
of the problem and the fact that most volunteer developers don't have
access to all those platforms. And yes, a build farm would help those
people. But it's not anyone's *responsibility* to provide that.
Indeed, the whole open source ecosystem is built on people seeing a
need and *putting their own effort into making it happen, and then
sharing the results*. You seem to think that your contribution should
be to nag people who are already providing freely of their time, to do
more, so that you don't have to.

> My intention was, to find a solution.

Well, it didn't come across that way.

From another post:
> I am volunteering for doing coordination work:
>  - communication
>  - layout of datastructures
>  - interchange of datastructures.
>  - no coding
>
> But we need at least ten people how say "I'm willing to help"

We don't need project managers for this task. We need people willing
to code, build servers, etc. If (and it's a big if) the management of
the work becomes too much of an overhead for those people, then (and
*only* then) they might appreciate some help with the administrative
tasks. But volunteering to co-ordinate resources that don't exist is
not useful. Nor is trying to find such resources from the pool of
people who are *already* doing plenty. If you want to find resources,
I suggest you look among people who are not yet contributing to the
packaging ecosystem.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] If you want wheel to be successful, provide a build server.

[Thomas Güttler]

Am 25.05.2016 um 15:56 schrieb Nick Coghlan:

On 25 May 2016 at 23:42, Thomas Güttler  wrote:

Am 25.05.2016 um 09:57 schrieb Alex Grönholm:


Amen to that, but who will pay for it? I imagine a great deal of
processing power would be required for this.
How do implementors of other languages handle this?


I talked with someone who is member of the python software foundation, and
he said that
money for projects like this is available. Of course this was no official
statement.


No, money for a project of this scale is not available (the person you
spoke to may have been thinking of PSF development grants, which
typically only cover 4-12 weeks of dedicated development work by a
single developer on a project with specific near term objectives, not
running on ongoing service that would dwarf PyPI itself in complexity)


Yes, running a permanent build-server might be too high regular expenses.

I think providing a self hostable build server which can be started with one 
command
would be such a project.

Regards,
  Thomas Güttler


--
Thomas Guettler http://www.thomas-guettler.de/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] If you want wheel to be successful, provide a build server.

[Thomas Güttler]

Am 25.05.2016 um 15:55 schrieb Paul Moore:

On 25 May 2016 at 14:42, Thomas Güttler  wrote:

Am 25.05.2016 um 09:57 schrieb Alex Grönholm:


Amen to that, but who will pay for it? I imagine a great deal of
processing power would be required for this.
How do implementors of other languages handle this?



I talked with someone who is member of the python software foundation, and
he said that
money for projects like this is available. Of course this was no official
statement.


The other aspect of this is who has sufficient time/expertise to set
something like this up? Are you volunteering to do this?


I am volunteering for doing coordination work:
 - communication
 - layout of datastructures
 - interchange of datastructures.
 - no coding

But we need at least ten people how say "I'm willing to help"

Regards,
  Thomas Güttler





--
Thomas Guettler http://www.thomas-guettler.de/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] If you want wheel to be successful, provide a build server.

[Thomas Güttler]

Am 25.05.2016 um 15:52 schrieb Nick Coghlan:

On 25 May 2016 at 17:13, Thomas Güttler  wrote:

If you want wheel to be successful, **provide a build server**.


Thomas, aside from that statement being demonstrably untrue (since the
wheel format has already proven to be wildly successful, even with
developers coping with Linux ABI fragmentation), an attitude of "give
me more free stuff, or your project will fail" is not an acceptable
tone to adopt on this list.

The contributors here are (mainly) volunteers working on
infrastructure provided by a public interest charity, not your
personal servants to be ordered about as you feel inclined.



You seem to be angry. Why?

I am just bringing information from A to B (from github issue
to this list).

I guess the author of psutil is just a volunteer like you are.

What is the problem? I think the problem is that
creating wheels for psutil is difficult.

I think we all agree on this.

If not, if you think it is easy, then please tell us.

My intention was, to find a solution.

Regards,
  Thomas Güttler





--
Thomas Guettler http://www.thomas-guettler.de/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] If you want wheel to be successful, provide a build server.

[Christopher Baines]

On 25/05/16 09:13, Thomas Güttler wrote:
> If you want wheel to be successful, **provide a build server**.
> 
> Quoting the author of psutil:
> 
> https://github.com/giampaolo/psutil/issues/824#issuecomment-221359292
> 
> {{{
> On Linux / Unix the only way you have to install psutil right now is via
> source / tarball. I don't want to provide wheels for Linux (or other
> UNIX platforms). I would have to cover all supported python versions (7)
> both 32 and 64 bits, meaning 14 extra packages to compile and upload on
> PYPI on every release. I do that for Windows because installing VS is an
> order of magnitude more difficult than installing gcc on Linux/UNIX but
> again: not willing to do extra work on that front (sorry).
> What you could do is create a wheel yourself with python setup.py build
> bdist_wheel by using the same python/arch version you have on the
> server, upload it on the server and install it with pip.
> }}}
> 
> What do you think?

This is something best left to operating systems/package managers, e.g.
Debian, Ubuntu, Fedora, RedHat, Guix(SD), Nix(OS). Contrary to the quote
above, I believe psutil is available from all of the projects above (in
a binary and source format).

These projects have the expressiveness to reason about packages that are
not just pure Python (e.g. that depend on GCC), and the software and
infrastructure (e.g. build servers) to manage this.

In my mind providing a build server is just not in scope for Python, or
PyPI.




signature.asc
Description: OpenPGP digital signature
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] If you want wheel to be successful, provide a build server.

[Nick Coghlan]

On 25 May 2016 at 17:13, Thomas Güttler  wrote:
> If you want wheel to be successful, **provide a build server**.

Thomas, aside from that statement being demonstrably untrue (since the
wheel format has already proven to be wildly successful, even with
developers coping with Linux ABI fragmentation), an attitude of "give
me more free stuff, or your project will fail" is not an acceptable
tone to adopt on this list.

The contributors here are (mainly) volunteers working on
infrastructure provided by a public interest charity, not your
personal servants to be ordered about as you feel inclined.

Regards,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] If you want wheel to be successful, provide a build server.

[Nick Coghlan]

On 25 May 2016 at 23:42, Thomas Güttler  wrote:
> Am 25.05.2016 um 09:57 schrieb Alex Grönholm:
>>
>> Amen to that, but who will pay for it? I imagine a great deal of
>> processing power would be required for this.
>> How do implementors of other languages handle this?
>
> I talked with someone who is member of the python software foundation, and
> he said that
> money for projects like this is available. Of course this was no official
> statement.

No, money for a project of this scale is not available (the person you
spoke to may have been thinking of PSF development grants, which
typically only cover 4-12 weeks of dedicated development work by a
single developer on a project with specific near term objectives, not
running on ongoing service that would dwarf PyPI itself in complexity)

Regards,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] If you want wheel to be successful, provide a build server.

[Paul Moore]

On 25 May 2016 at 14:42, Thomas Güttler  wrote:
> Am 25.05.2016 um 09:57 schrieb Alex Grönholm:
>>
>> Amen to that, but who will pay for it? I imagine a great deal of
>> processing power would be required for this.
>> How do implementors of other languages handle this?
>
>
> I talked with someone who is member of the python software foundation, and
> he said that
> money for projects like this is available. Of course this was no official
> statement.

The other aspect of this is who has sufficient time/expertise to set
something like this up? Are you volunteering to do this?
Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] If you want wheel to be successful, provide a build server.

[Thomas Güttler]

Am 25.05.2016 um 09:57 schrieb Alex Grönholm:

Amen to that, but who will pay for it? I imagine a great deal of processing 
power would be required for this.
How do implementors of other languages handle this?


I talked with someone who is member of the python software foundation, and he 
said that
money for projects like this is available. Of course this was no official 
statement.

Regards,
  Thomas Güttler

--
Thomas Guettler http://www.thomas-guettler.de/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] If you want wheel to be successful, provide a build server.

[Noah Kantrowitz]

> On May 25, 2016, at 12:13 AM, Thomas Güttler  
> wrote:
> 
> If you want wheel to be successful, **provide a build server**.
> 
> Quoting the author of psutil:
> 
> https://github.com/giampaolo/psutil/issues/824#issuecomment-221359292
> 
> {{{
> On Linux / Unix the only way you have to install psutil right now is via 
> source / tarball. I don't want to provide wheels for Linux (or other UNIX 
> platforms). I would have to cover all supported python versions (7) both 32 
> and 64 bits, meaning 14 extra packages to compile and upload on PYPI on every 
> release. I do that for Windows because installing VS is an order of magnitude 
> more difficult than installing gcc on Linux/UNIX but again: not willing to do 
> extra work on that front (sorry).
> What you could do is create a wheel yourself with python setup.py build 
> bdist_wheel by using the same python/arch version you have on the server, 
> upload it on the server and install it with pip.
> }}}
> 
> What do you think?

The problems haven't really changed every time someone brings this up. Running 
untrusted code from the internet isn't impossible (eg. Travis, Heroku, Lambda) 
but it requires serious care and feeding at a scale we don't currently have the 
resources for. Until something in that equation changes, the best we can do it 
try to piggyback on an existing sandbox environment like Travis.

--Noah




signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] If you want wheel to be successful, provide a build server.

[Paul Moore]

On 25 May 2016 at 08:57, Ionel Cristian Mărieș  wrote:
> He may accept a PR with Travis configuration to build the wheels though ...
> Using that manylinux docker image should be easy enough on Travis.

A PR to the packaging user Guide explaining how to build Linux
(manylinux) wheels using Travis would be a worthwhile addition (it
would sit nicely alongside
https://packaging.python.org/en/latest/appveyor/ which explains how to
do the same for Windows using Appveyor).

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] If you want wheel to be successful, provide a build server.

[Ionel Cristian Mărieș via Distutils-SIG]

On Wed, May 25, 2016 at 10:13 AM, Thomas Güttler <
guettl...@thomas-guettler.de> wrote:

> I do that for Windows because installing VS is an order of magnitude more
> difficult than installing gcc on Linux/UNIX but again: not willing to do
> extra work on that front (sorry).


He may accept a PR with Travis configuration to build the wheels though ...
Using that manylinux docker image should be easy enough on Travis.​

​On the other hand, there might be weird dependencies and the wheel don't
build well on the manylinux container - so there's still some extra effort​
for the maintainer.

A simpler package would probably make a better discussion.

PS. To  build all the manylinux wheels for a package you only need to run
something like:
docker run -itv $(pwd):/code quay.io/pypa/manylinux1_x86_64 bash -c 'set
-eux; cd code; rm -rf wheelhouse; for variant in /opt/python/*; do rm -rf
dist build *.egg-info && $variant/bin/python setup.py clean --all
bdist_wheel; auditwheel repair dist/*.whl; done; rm -rf dist build
*.egg-info'

​


Thanks,
-- Ionel Cristian Mărieș, http://blog.ionelmc.ro
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] If you want wheel to be successful, provide a build server.

[Alex Grönholm]

Amen to that, but who will pay for it? I imagine a great deal of 
processing power would be required for this.

How do implementors of other languages handle this?

25.05.2016, 10:13, Thomas Güttler kirjoitti:

If you want wheel to be successful, **provide a build server**.

Quoting the author of psutil:

https://github.com/giampaolo/psutil/issues/824#issuecomment-221359292

{{{
On Linux / Unix the only way you have to install psutil right now is 
via source / tarball. I don't want to provide wheels for Linux (or 
other UNIX platforms). I would have to cover all supported python 
versions (7) both 32 and 64 bits, meaning 14 extra packages to compile 
and upload on PYPI on every release. I do that for Windows because 
installing VS is an order of magnitude more difficult than installing 
gcc on Linux/UNIX but again: not willing to do extra work on that 
front (sorry).
What you could do is create a wheel yourself with python setup.py 
build bdist_wheel by using the same python/arch version you have on 
the server, upload it on the server and install it with pip.

}}}

What do you think?

Regards,
  Thomas Güttler



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] If you want wheel to be successful, provide a build server.

[Thomas Güttler]

If you want wheel to be successful, **provide a build server**.

Quoting the author of psutil:

https://github.com/giampaolo/psutil/issues/824#issuecomment-221359292

{{{
On Linux / Unix the only way you have to install psutil right now is via source / tarball. I don't want to provide 
wheels for Linux (or other UNIX platforms). I would have to cover all supported python versions (7) both 32 and 64 bits, 
meaning 14 extra packages to compile and upload on PYPI on every release. I do that for Windows because installing VS is 
an order of magnitude more difficult than installing gcc on Linux/UNIX but again: not willing to do extra work on that 
front (sorry).
What you could do is create a wheel yourself with python setup.py build bdist_wheel by using the same python/arch 
version you have on the server, upload it on the server and install it with pip.

}}}

What do you think?

Regards,
  Thomas Güttler

--
Thomas Guettler http://www.thomas-guettler.de/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig