Re: [Distutils] warning about potential problem for wheels

[Chris Barker]

On Wed, Oct 14, 2015 at 1:16 PM, Oscar Benjamin 
wrote:

> Numpy's Windows installer bundles several BLAS binaries with different
> levels of SSE and this was the initial reason for not providing Windows
> wheels.
>
indeed -- but IIRC, SSE2 made a pretty big difference, but SSE3 not so much
-- though, of course, who knows how fabulously useful some future extension
might be...


> The problem is being solved though by switching from ATLAS to OpenBLAS
> which selects different levels of SSE at runtime.
>
> Maybe that approach (runtime machine code selection) is the only way to
> marry the needs of packaging with the desire to fully utilise CPU
> capabilities. It keeps the packaging side simple at the expense of pushing
> the complexity onto the project authors. Though it's probably only viable
> for something like a BLAS library which would often  contain a load of hand
> crafted assembly code anyway.
>
only viable ,and probably only necessary... I'd be really surprised if SSE2
made any notabel difference in regular old python code.

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] warning about potential problem for wheels

[Oscar Benjamin]

On 14 Oct 2015 19:00, "Chris Barker"  wrote:
>
> On Wed, Oct 14, 2015 at 9:54 AM, Antoine Pitrou 
wrote:
>>
>> > IS that the case:
>> > """
>> > Note that my recently retired computer was 64 bit and had SSE but
didn't
>> > have SSE2 (I'm fairly sure - CPU was some budget AMD model)
>> > """
>> >
>> > granted, such machines are probably really really rare, but maybe it
does
>> > matter for 64 bit, too?
>>
>> Unless I'm mistaken, SSE2 is part of the spec for x86-64 (spec which
>> was originally devised by AMD), so I'm a bit skeptical about a
>> SSE2-less 64-bit CPU.  Do you have any reference?
>
>
> That was a quote from this thread... I have no idea beyond that.

I wrote that but now I think about it Antoine is right. SSE2 is fully
supported on all x86-64 CPUs. I must have been confusing my old home
computer with my old work computer (which was 32 bit and ran XP). No way to
check now though...

The problem with SSE2 may go away soon but then we have the problem with
SSE3 and so on. Most of us have CPUs that support instruction sets beyond
those used in the lowest common denominator builds of Python provided in
the Windows binaries (and distro binaries etc). Likewise for extension
modules on PyPI.

Numpy's Windows installer bundles several BLAS binaries with different
levels of SSE and this was the initial reason for not providing Windows
wheels. The problem is being solved though by switching from ATLAS to
OpenBLAS which selects different levels of SSE at runtime.

Maybe that approach (runtime machine code selection) is the only way to
marry the needs of packaging with the desire to fully utilise CPU
capabilities. It keeps the packaging side simple at the expense of pushing
the complexity onto the project authors. Though it's probably only viable
for something like a BLAS library which would often  contain a load of hand
crafted assembly code anyway.

--
Oscar
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] warning about potential problem for wheels

[Chris Barker]

On Wed, Oct 14, 2015 at 9:54 AM, Antoine Pitrou  wrote:

> > IS that the case:
> > """
> > Note that my recently retired computer was 64 bit and had SSE but didn't
> > have SSE2 (I'm fairly sure - CPU was some budget AMD model)
> > """
> >
> > granted, such machines are probably really really rare, but maybe it does
> > matter for 64 bit, too?
>
> Unless I'm mistaken, SSE2 is part of the spec for x86-64 (spec which
> was originally devised by AMD), so I'm a bit skeptical about a
> SSE2-less 64-bit CPU.  Do you have any reference?


That was a quote from this thread... I have no idea beyond that.


> > > does this even matter?
> > > 32-bit builds on x86 generally bring you poorer performance by
> > > themselves,
> >
> > If a user has a 32 bit machine, they have no choice -- we could argue
> that
> > anyone for whom performance matters probably isn't running an old, cheap
> > machine, but still...
>
> The actual question is whether we want to introduce a significant
> amount of complexity and overhead for packagers and distributors, for
> the benefit of an extremely small users demography that will probably
> disappear altogether in a couple of years.
>

Actually, I thought you were advocating the opposite:

SSE2 on 64bit -- cause 64 bit machines without SSE2 are really rare, if
they exist at all.

no SSE2 on 32 bit -- because anyone to whom performance really matters
won't be running an old 32bit machine.

Seems reasonable to me.

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] warning about potential problem for wheels

[Antoine Pitrou]

On Wed, 14 Oct 2015 09:25:43 -0700
Chris Barker  wrote:
> On Wed, Oct 14, 2015 at 3:40 AM, Antoine Pitrou  wrote:
> 
> > On Tue, 13 Oct 2015 09:59:05 -0700
> > Chris Barker  wrote:
> > >
> > > So even is SSE2 provides little for Python itself, in the usual case,
> > we'll
> > > see performance hits n compiled extensions that are not compiled by
> > > particularly smart people.
> >
> > Since the question is only for 32-bit builds,
> 
> IS that the case:
> """
> Note that my recently retired computer was 64 bit and had SSE but didn't
> have SSE2 (I'm fairly sure - CPU was some budget AMD model)
> """
> 
> granted, such machines are probably really really rare, but maybe it does
> matter for 64 bit, too?

Unless I'm mistaken, SSE2 is part of the spec for x86-64 (spec which
was originally devised by AMD), so I'm a bit skeptical about a
SSE2-less 64-bit CPU.  Do you have any reference?

> > does this even matter?
> > 32-bit builds on x86 generally bring you poorer performance by
> > themselves,
> 
> If a user has a 32 bit machine, they have no choice -- we could argue that
> anyone for whom performance matters probably isn't running an old, cheap
> machine, but still...

The actual question is whether we want to introduce a significant
amount of complexity and overhead for packagers and distributors, for
the benefit of an extremely small users demography that will probably
disappear altogether in a couple of years.

Regards

Antoine.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] warning about potential problem for wheels

[Chris Barker]

On Wed, Oct 14, 2015 at 3:40 AM, Antoine Pitrou  wrote:

> On Tue, 13 Oct 2015 09:59:05 -0700
> Chris Barker  wrote:
> >
> > So even is SSE2 provides little for Python itself, in the usual case,
> we'll
> > see performance hits n compiled extensions that are not compiled by
> > particularly smart people.
>
> Since the question is only for 32-bit builds,


IS that the case:
"""
Note that my recently retired computer was 64 bit and had SSE but didn't
have SSE2 (I'm fairly sure - CPU was some budget AMD model)
"""

granted, such machines are probably really really rare, but maybe it does
matter for 64 bit, too?


> does this even matter?
> 32-bit builds on x86 generally bring you poorer performance by
> themselves,


If a user has a 32 bit machine, they have no choice -- we could argue that
anyone for whom performance matters probably isn't running an old, cheap
machine, but still...


>  If you want better performance (or want
> to efficiently address 3+GB of RAM, which is an extremely common
> situation nowadays), you probably want 64-bit builds.
>

I thought that wasn't the case (well it certainly is if you want to access
that memory), but for raw performance, 64 bit is pushing a lot more memory,
in the form of larger pointers, around -- so is sometimes slower. I'm sure
it depends on the use-case, as it always does

Also a note -- I'm lost on the details, but the most recent numpy release
required SSE2:

"""
* Compiling with msvc9 or msvc10 for 32 bit Windows now requires SSE2.
  This was the easiest fix for what looked to be some miscompiled code when
  SSE2 was not used. If you need to compile for 32 bit Windows systems
  without SSE2 support, mingw32 should still work.
"""

and that's not py3.5 (msvc9 is the py2.7 compiler, and msvc10 is py3.4,
yes???)

-Chris











>
> Regards
>
> Antoine.
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] warning about potential problem for wheels

[Antoine Pitrou]

On Tue, 13 Oct 2015 09:59:05 -0700
Chris Barker  wrote:
> 
> So even is SSE2 provides little for Python itself, in the usual case, we'll
> see performance hits n compiled extensions that are not compiled by
> particularly smart people.

Since the question is only for 32-bit builds, does this even matter?
32-bit builds on x86 generally bring you poorer performance by
themselves, as the CPU has only access to 8 general-purpose registers
(instead of 16) in that mode.  If you want better performance (or want
to efficiently address 3+GB of RAM, which is an extremely common
situation nowadays), you probably want 64-bit builds.

Regards

Antoine.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] warning about potential problem for wheels

[Chris Barker - NOAA Federal]

But even if so do we WANT to now be supporting 4 Windows platforms?

win32
win32_sse2
win64
win64_sse2

and then, what about sse3, etc???

Longer term, more platforms (particularly more platforms that really only
differ by some compiler flags) shouldn’t be a very big deal, because my
dream for PyPI is that authors upload a source package and PyPI builds all
their wheels for them (optionally of course).

That would be pretty nice. I'm still a bit wary though -- the more
platforms you have, the more you have to test and debug

Testing, off course would be part of the automated build. But debugging
could get painful ...

CHB



-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] warning about potential problem for wheels

[Donald Stufft]

On October 13, 2015 at 1:07:44 PM, Chris Barker (chris.bar...@noaa.gov) wrote:
yeah, that's the trick with trying to shoehorn dependencies liek this into 
platform tags.

But even if so do we WANT to now be supporting 4 Windows platforms?

win32
win32_sse2
win64
win64_sse2

and then, what about sse3, etc???
Longer term, more platforms (particularly more platforms that really only 
differ by some compiler flags) shouldn’t be a very big deal, because my dream 
for PyPI is that authors upload a source package and PyPI builds all their 
wheels for them (optionally of course).

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] warning about potential problem for wheels

[Chris Barker]

On Sun, Oct 11, 2015 at 11:18 AM, Steve Dower 
wrote:

>  And it's also not binding on redistributors or package builders, who are
> free to add more restrictive requirements.
>

not binding, but presumably a simple:

setup.py build

with distutils (or setuptools) will NOT use SSE2 in this case? That's the
beauty of distutils of course, that is builds extensions with all the same
settings as the host python.

So even is SSE2 provides little for Python itself, in the usual case, we'll
see performance hits n compiled extensions that are not compiled by
particularly smart people.

Also -- this thread started with "I can build Python3 myself, but now I
can't install arbitrary binary wheels" -- if the goal is to fix that, then
we don't WANT people tweaking things to build their wheels with SSE2 -- it
would just get uglier.

I certanly thought the "official" policy was that binary wheels put up on
PyPi be (fully) compatible with the python.org build(s).

What I'm getting at is that a decision to not use SSE2 in the main python
build is a decision to not use it for any PyPi packages -- which could be a
Bad Thing.


> As far as identifying these sort of requirements in packages, I don't
> think they really make sense as part of a platform tag as a non-SSE build
> is fully compatible with an SSE build, but as part of version ordering it
> would be nice to be able to prefer certain tagged builds without excluding
> ones without the tag (e.g. if my platform is win32_sse2 and there's only a
> win32 tagged build, I still want it - how we formalize this is probably
> very complex though.
>

yeah, that's the trick with trying to shoehorn dependencies liek this into
platform tags.

But even if so do we WANT to now be supporting 4 Windows platforms?

win32
win32_sse2
win64
win64_sse2

and then, what about sse3, etc???

My thoughts -- pick one, and either:

* SSE2 is a requirement for python3.5 on Windows
  - then schools and the like can build their own Python3.5, AND their own
package repository
- on the one hand, institutions should be able to do this
- on the other hand, the same institutions that NEED to do this (nes
with old hardware),. are probably short on other resources, like admins
savvy enough to be setting up build systems...

* SSE2 is NOT used for the Windows builds.
  - then all packages on PyPi should also not use SSE2
  - Other organizations, like Anaconda, Enthought, etc can build their own
python and dependent packages can be build against that.
- At least these folks DO have the resources to support that.

-CHB






> Also, my platform tag is probably 100 chars long with everything I may
> want to represent...)
>
> Cheers,
> Steve
>
> Top-posted from my Windows Phone
> ------
> From: Oscar Benjamin 
> Sent: ‎10/‎11/‎2015 10:43
> To: Antoine Pitrou ; Distutils-Sig@python.org
> Subject: Re: [Distutils] warning about potential problem for wheels
>
> On Sun, 11 Oct 2015 17:44 Antoine Pitrou  wrote:
>
> On Sun, 11 Oct 2015 08:07:30 -0700
> Steve Dower  wrote:
> >
> > This does only affect 32-bit builds, so now I'm thinking about the
> > possibility of treating those as highly compatible while the 64-bit
> > ones get better performance treatment, though I'm not sure how that
> > could actually play out. It may help remove some of the questions
> > about which one to use though.
>
> That sounds reasonable to me. I don't know Windows very much, but are
> there still many people using 32-bit Windows these days (on x86, I
> mean)?
>
>
>
> I don't know but I think it makes sense to follow Windows' lead. So if 3.5
> supports Vista and Vista doesn't require SSE2 then CPython shouldn't
> either. If 3.6 or whatever drops support for Vista and if Windows 7
> requires SSE2 then CPython can require it too. I assume this what happens
> with the OSX binaries.
>
> Note that my recently retired computer was 64 bit and had SSE but didn't
> have SSE2 (I'm fairly sure - CPU was some budget AMD model). Also after
> SSE2 we have SSE3 etc and I've seen no indication that x86-64 manufacturers
> are going to stop adding new instructions. So this general issue isn't
> limited to 32 bit hardware and won't be solved by special casing that. I
> think it makes sense to have a general policy for architectures that will
> be supported by the official build in future.
>
> --
> Oscar
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] warning about potential problem for wheels

[Steve Dower]

The general rule is the minimum requirements of all supported operating 
systems, so in this case requiring SSE is a bug.

Note that this doesn't prevent us from having SSE optimized code paths, as long 
as there is a fallback to one that is more compatible. And it's also not 
binding on redistributors or package builders, who are free to add more 
restrictive requirements.

As far as identifying these sort of requirements in packages, I don't think 
they really make sense as part of a platform tag as a non-SSE build is fully 
compatible with an SSE build, but as part of version ordering it would be nice 
to be able to prefer certain tagged builds without excluding ones without the 
tag (e.g. if my platform is win32_sse2 and there's only a win32 tagged build, I 
still want it - how we formalize this is probably very complex though. Also, my 
platform tag is probably 100 chars long with everything I may want to 
represent...)

Cheers,
Steve

Top-posted from my Windows Phone

-Original Message-
From: "Oscar Benjamin" 
Sent: ‎10/‎11/‎2015 10:43
To: "Antoine Pitrou" ; "Distutils-Sig@python.org" 

Subject: Re: [Distutils] warning about potential problem for wheels

On Sun, 11 Oct 2015 17:44 Antoine Pitrou  wrote:
On Sun, 11 Oct 2015 08:07:30 -0700
Steve Dower  wrote:
>
> This does only affect 32-bit builds, so now I'm thinking about the
> possibility of treating those as highly compatible while the 64-bit
> ones get better performance treatment, though I'm not sure how that
> could actually play out. It may help remove some of the questions
> about which one to use though.
That sounds reasonable to me. I don't know Windows very much, but are
there still many people using 32-bit Windows these days (on x86, I
mean)?




I don't know but I think it makes sense to follow Windows' lead. So if 3.5 
supports Vista and Vista doesn't require SSE2 then CPython shouldn't either. If 
3.6 or whatever drops support for Vista and if Windows 7 requires SSE2 then 
CPython can require it too. I assume this what happens with the OSX binaries.
Note that my recently retired computer was 64 bit and had SSE but didn't have 
SSE2 (I'm fairly sure - CPU was some budget AMD model). Also after SSE2 we have 
SSE3 etc and I've seen no indication that x86-64 manufacturers are going to 
stop adding new instructions. So this general issue isn't limited to 32 bit 
hardware and won't be solved by special casing that. I think it makes sense to 
have a general policy for architectures that will be supported by the official 
build in future.
--
Oscar___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] warning about potential problem for wheels

[Oscar Benjamin]

On Sun, 11 Oct 2015 17:44 Antoine Pitrou  wrote:

On Sun, 11 Oct 2015 08:07:30 -0700
Steve Dower  wrote:
>
> This does only affect 32-bit builds, so now I'm thinking about the
> possibility of treating those as highly compatible while the 64-bit
> ones get better performance treatment, though I'm not sure how that
> could actually play out. It may help remove some of the questions
> about which one to use though.

That sounds reasonable to me. I don't know Windows very much, but are
there still many people using 32-bit Windows these days (on x86, I
mean)?



I don't know but I think it makes sense to follow Windows' lead. So if 3.5
supports Vista and Vista doesn't require SSE2 then CPython shouldn't
either. If 3.6 or whatever drops support for Vista and if Windows 7
requires SSE2 then CPython can require it too. I assume this what happens
with the OSX binaries.

Note that my recently retired computer was 64 bit and had SSE but didn't
have SSE2 (I'm fairly sure - CPU was some budget AMD model). Also after
SSE2 we have SSE3 etc and I've seen no indication that x86-64 manufacturers
are going to stop adding new instructions. So this general issue isn't
limited to 32 bit hardware and won't be solved by special casing that. I
think it makes sense to have a general policy for architectures that will
be supported by the official build in future.

--
Oscar
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] warning about potential problem for wheels

[Antoine Pitrou]

On Sun, 11 Oct 2015 08:07:30 -0700
Steve Dower  wrote:
> 
> This does only affect 32-bit builds, so now I'm thinking about the
> possibility of treating those as highly compatible while the 64-bit
> ones get better performance treatment, though I'm not sure how that
> could actually play out. It may help remove some of the questions
> about which one to use though.

That sounds reasonable to me. I don't know Windows very much, but are
there still many people using 32-bit Windows these days (on x86, I
mean)?

Regards

Antoine.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] warning about potential problem for wheels

[Donald Stufft]

Okay. Thanks (and Oscar too). I'm not really that great at C and compiled stuff 
so I wasn't sure if it would have any negative effects like that.  Seems like a 
reasonable change. 

Sent from my iPhone

> On Oct 11, 2015, at 11:07 AM, Steve Dower  wrote:
> 
> Yes. SSE never makes it into the module ABI, it's just an implementation 
> concern. (I believe it may be used across function/module boundaries 
> internally when the compiler can prove that it's inaccessible from the 
> outside - a more space effective form of inlining.)
> 
> There will be some 3.5.0-built wheels that won't work on some machines 
> running 3.5.1 without SSE, and some wheels will see a performance decrease 
> when built with 3.5.1 vs 3.5.0 (but not compared to 3.4). These latter ones 
> weren't expecting the perf increase anyway, so compatibility is likely there 
> biggest concern.
> 
> This does only affect 32-bit builds, so now I'm thinking about the 
> possibility of treating those as highly compatible while the 64-bit ones get 
> better performance treatment, though I'm not sure how that could actually 
> play out. It may help remove some of the questions about which one to use 
> though.
> 
> Cheers,
> Steve
> 
> Top-posted from my Windows Phone
> From: Donald Stufft
> Sent: ‎10/‎11/‎2015 7:31
> To: Steve Dower
> Cc: distutils-sig; Laura Creighton
> Subject: Re: [Distutils] warning about potential problem for wheels
> 
> Will something built against 3.5.0 with SSE work on 3.5.1 without SSE? What 
> about the inverse?
> 
> Sent from my iPhone
> 
>> On Oct 11, 2015, at 10:17 AM, Steve Dower  wrote:
>> 
>> An extra data point is that we've had exactly one report of Python 3.5 not 
>> working due to lack of SSE, and that person was also on Windows XP (so zero 
>> reports on supported platforms).
>> 
>> That said, I should probably just fix 3.5.1 to not use SSE for core or 
>> distutils builds. I doubt there was a huge performance increase due to it 
>> (mainly in memcpy I'd assume).
>> 
>> Cheers,
>> Steve
>> 
>> Top-posted from my Windows Phone
>> From: Nathaniel Smith
>> Sent: ‎10/‎10/‎2015 16:11
>> To: Laura Creighton
>> Cc: distutils-sig
>> Subject: Re: [Distutils] warning about potential problem for wheels
>> 
>> On Oct 10, 2015 3:37 PM, "Laura Creighton"  wrote:
>> >
>> > In a message of Sat, 10 Oct 2015 21:52:58 -, Oscar Benjamin writes:
>> >
>> > >Really this is just a case of an unsupported platform. It's unfortunate
>> > >that CPython doesn't properly support this hardware but I think it's
>> > >reasonable that if you have to build your interpreter from source then you
>> > >have to build your extension modules as well.
>> >
>> > Alas that there is no easy way to detect.  The situation I am
>> > imagining is where the administrators of a school build pythons for
>> > the students to run on their obsolete hardware, and then the poor
>> > students don't understand why pip doesn't work.  But I suppose we
>> > will just get to deal with that problem when and if it happens.
>> 
>> In case some numbers help: the numerical community has been tracking the 
>> deployment of SSE2 to decide when to switch over for numpy/scipy builds, and 
>> at this point what I hear is:
>> - ~0.5% of Firefox crashes are on machines that are missing SSE2
>> - <0.1% of machines with Steam installed are missing SSE2
>> 
>> I'm not sure what python distributions like Anaconda or Activestate are 
>> doing wrt SSE2, but even if their builds do require SSE2 then their build 
>> tooling might provide a quick way to generate a whole installable 
>> environment with custom build options for targeting older systems. 
>> (Continuum as of today still hasn't released build scripts for the bottom of 
>> their stack -- python/numpy/etc. -- but they've claimed willingness to do so 
>> and there's increasing calls to make a "centos" version of Anaconda. And all 
>> the tooling beyond that -- e.g. the actual package manager -- is FOSS.)
>> 
>> -n
>> 
>> ___
>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>> https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] warning about potential problem for wheels

[Steve Dower]

Yes. SSE never makes it into the module ABI, it's just an implementation 
concern. (I believe it may be used across function/module boundaries internally 
when the compiler can prove that it's inaccessible from the outside - a more 
space effective form of inlining.)

There will be some 3.5.0-built wheels that won't work on some machines running 
3.5.1 without SSE, and some wheels will see a performance decrease when built 
with 3.5.1 vs 3.5.0 (but not compared to 3.4). These latter ones weren't 
expecting the perf increase anyway, so compatibility is likely there biggest 
concern.

This does only affect 32-bit builds, so now I'm thinking about the possibility 
of treating those as highly compatible while the 64-bit ones get better 
performance treatment, though I'm not sure how that could actually play out. It 
may help remove some of the questions about which one to use though.

Cheers,
Steve

Top-posted from my Windows Phone

-Original Message-
From: "Donald Stufft" 
Sent: ‎10/‎11/‎2015 7:31
To: "Steve Dower" 
Cc: "distutils-sig" ; "Laura Creighton" 

Subject: Re: [Distutils] warning about potential problem for wheels

Will something built against 3.5.0 with SSE work on 3.5.1 without SSE? What 
about the inverse?

Sent from my iPhone

On Oct 11, 2015, at 10:17 AM, Steve Dower  wrote:


An extra data point is that we've had exactly one report of Python 3.5 not 
working due to lack of SSE, and that person was also on Windows XP (so zero 
reports on supported platforms).

That said, I should probably just fix 3.5.1 to not use SSE for core or 
distutils builds. I doubt there was a huge performance increase due to it 
(mainly in memcpy I'd assume).

Cheers,
Steve

Top-posted from my Windows Phone


From: Nathaniel Smith
Sent: ‎10/‎10/‎2015 16:11
To: Laura Creighton
Cc: distutils-sig
Subject: Re: [Distutils] warning about potential problem for wheels


On Oct 10, 2015 3:37 PM, "Laura Creighton"  wrote:
>
> In a message of Sat, 10 Oct 2015 21:52:58 -, Oscar Benjamin writes:
>
> >Really this is just a case of an unsupported platform. It's unfortunate
> >that CPython doesn't properly support this hardware but I think it's
> >reasonable that if you have to build your interpreter from source then you
> >have to build your extension modules as well.
>
> Alas that there is no easy way to detect.  The situation I am
> imagining is where the administrators of a school build pythons for
> the students to run on their obsolete hardware, and then the poor
> students don't understand why pip doesn't work.  But I suppose we
> will just get to deal with that problem when and if it happens.
In case some numbers help: the numerical community has been tracking the 
deployment of SSE2 to decide when to switch over for numpy/scipy builds, and at 
this point what I hear is:
- ~0.5% of Firefox crashes are on machines that are missing SSE2
- <0.1% of machines with Steam installed are missing SSE2
I'm not sure what python distributions like Anaconda or Activestate are doing 
wrt SSE2, but even if their builds do require SSE2 then their build tooling 
might provide a quick way to generate a whole installable environment with 
custom build options for targeting older systems. (Continuum as of today still 
hasn't released build scripts for the bottom of their stack -- 
python/numpy/etc. -- but they've claimed willingness to do so and there's 
increasing calls to make a "centos" version of Anaconda. And all the tooling 
beyond that -- e.g. the actual package manager -- is FOSS.)
-n
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] warning about potential problem for wheels

[Oscar Benjamin]

On Sun, 11 Oct 2015 15:31 Donald Stufft  wrote:

Will something built against 3.5.0 with SSE work on 3.5.1 without SSE? What
about the inverse?


It should be fine either way as long as the CPU can handle the particular
instructions used. X86 is backward compatible like that so unless the
compiler does something funny when the SSE option is enabled it should be
fine.

--
Oscar
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] warning about potential problem for wheels

[Donald Stufft]

Will something built against 3.5.0 with SSE work on 3.5.1 without SSE? What 
about the inverse?

Sent from my iPhone

> On Oct 11, 2015, at 10:17 AM, Steve Dower  wrote:
> 
> An extra data point is that we've had exactly one report of Python 3.5 not 
> working due to lack of SSE, and that person was also on Windows XP (so zero 
> reports on supported platforms).
> 
> That said, I should probably just fix 3.5.1 to not use SSE for core or 
> distutils builds. I doubt there was a huge performance increase due to it 
> (mainly in memcpy I'd assume).
> 
> Cheers,
> Steve
> 
> Top-posted from my Windows Phone
> From: Nathaniel Smith
> Sent: ‎10/‎10/‎2015 16:11
> To: Laura Creighton
> Cc: distutils-sig
> Subject: Re: [Distutils] warning about potential problem for wheels
> 
> On Oct 10, 2015 3:37 PM, "Laura Creighton"  wrote:
> >
> > In a message of Sat, 10 Oct 2015 21:52:58 -, Oscar Benjamin writes:
> >
> > >Really this is just a case of an unsupported platform. It's unfortunate
> > >that CPython doesn't properly support this hardware but I think it's
> > >reasonable that if you have to build your interpreter from source then you
> > >have to build your extension modules as well.
> >
> > Alas that there is no easy way to detect.  The situation I am
> > imagining is where the administrators of a school build pythons for
> > the students to run on their obsolete hardware, and then the poor
> > students don't understand why pip doesn't work.  But I suppose we
> > will just get to deal with that problem when and if it happens.
> 
> In case some numbers help: the numerical community has been tracking the 
> deployment of SSE2 to decide when to switch over for numpy/scipy builds, and 
> at this point what I hear is:
> - ~0.5% of Firefox crashes are on machines that are missing SSE2
> - <0.1% of machines with Steam installed are missing SSE2
> 
> I'm not sure what python distributions like Anaconda or Activestate are doing 
> wrt SSE2, but even if their builds do require SSE2 then their build tooling 
> might provide a quick way to generate a whole installable environment with 
> custom build options for targeting older systems. (Continuum as of today 
> still hasn't released build scripts for the bottom of their stack -- 
> python/numpy/etc. -- but they've claimed willingness to do so and there's 
> increasing calls to make a "centos" version of Anaconda. And all the tooling 
> beyond that -- e.g. the actual package manager -- is FOSS.)
> 
> -n
> 
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] warning about potential problem for wheels

[Steve Dower]

An extra data point is that we've had exactly one report of Python 3.5 not 
working due to lack of SSE, and that person was also on Windows XP (so zero 
reports on supported platforms).

That said, I should probably just fix 3.5.1 to not use SSE for core or 
distutils builds. I doubt there was a huge performance increase due to it 
(mainly in memcpy I'd assume).

Cheers,
Steve

Top-posted from my Windows Phone

-Original Message-
From: "Nathaniel Smith" 
Sent: ‎10/‎10/‎2015 16:11
To: "Laura Creighton" 
Cc: "distutils-sig" 
Subject: Re: [Distutils] warning about potential problem for wheels

On Oct 10, 2015 3:37 PM, "Laura Creighton"  wrote:
>
> In a message of Sat, 10 Oct 2015 21:52:58 -, Oscar Benjamin writes:
>
> >Really this is just a case of an unsupported platform. It's unfortunate
> >that CPython doesn't properly support this hardware but I think it's
> >reasonable that if you have to build your interpreter from source then you
> >have to build your extension modules as well.
>
> Alas that there is no easy way to detect.  The situation I am
> imagining is where the administrators of a school build pythons for
> the students to run on their obsolete hardware, and then the poor
> students don't understand why pip doesn't work.  But I suppose we
> will just get to deal with that problem when and if it happens.
In case some numbers help: the numerical community has been tracking the 
deployment of SSE2 to decide when to switch over for numpy/scipy builds, and at 
this point what I hear is:
- ~0.5% of Firefox crashes are on machines that are missing SSE2
- <0.1% of machines with Steam installed are missing SSE2
I'm not sure what python distributions like Anaconda or Activestate are doing 
wrt SSE2, but even if their builds do require SSE2 then their build tooling 
might provide a quick way to generate a whole installable environment with 
custom build options for targeting older systems. (Continuum as of today still 
hasn't released build scripts for the bottom of their stack -- 
python/numpy/etc. -- but they've claimed willingness to do so and there's 
increasing calls to make a "centos" version of Anaconda. And all the tooling 
beyond that -- e.g. the actual package manager -- is FOSS.)
-n___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] warning about potential problem for wheels

[Nathaniel Smith]

On Oct 10, 2015 3:37 PM, "Laura Creighton"  wrote:
>
> In a message of Sat, 10 Oct 2015 21:52:58 -, Oscar Benjamin writes:
>
> >Really this is just a case of an unsupported platform. It's unfortunate
> >that CPython doesn't properly support this hardware but I think it's
> >reasonable that if you have to build your interpreter from source then
you
> >have to build your extension modules as well.
>
> Alas that there is no easy way to detect.  The situation I am
> imagining is where the administrators of a school build pythons for
> the students to run on their obsolete hardware, and then the poor
> students don't understand why pip doesn't work.  But I suppose we
> will just get to deal with that problem when and if it happens.

In case some numbers help: the numerical community has been tracking the
deployment of SSE2 to decide when to switch over for numpy/scipy builds,
and at this point what I hear is:
- ~0.5% of Firefox crashes are on machines that are missing SSE2
- <0.1% of machines with Steam installed are missing SSE2

I'm not sure what python distributions like Anaconda or Activestate are
doing wrt SSE2, but even if their builds do require SSE2 then their build
tooling might provide a quick way to generate a whole installable
environment with custom build options for targeting older systems.
(Continuum as of today still hasn't released build scripts for the bottom
of their stack -- python/numpy/etc. -- but they've claimed willingness to
do so and there's increasing calls to make a "centos" version of Anaconda.
And all the tooling beyond that -- e.g. the actual package manager -- is
FOSS.)

-n
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] warning about potential problem for wheels

[Oscar Benjamin]

On Sat, 10 Oct 2015 23:37 Laura Creighton  wrote:

In a message of Sat, 10 Oct 2015 21:52:58 -, Oscar Benjamin writes:

>Really this is just a case of an unsupported platform. It's unfortunate
>that CPython doesn't properly support this hardware but I think it's
>reasonable that if you have to build your interpreter from source then you
>have to build your extension modules as well.

Alas that there is no easy way to detect.  The situation I am
imagining is where the administrators of a school build pythons for
the students to run on their obsolete hardware, and then the poor
students don't understand why pip doesn't work.  But I suppose we
will just get to deal with that problem when and if it happens.



Does it sound plausible to you that a school would build their own Pythons?
I only know a few schools and I'd be very surprised if this happened at one
of them but I guess there's a lot of schools in the world...

The administrators at my daughter's school don't even understand how to put
text into an email let alone install compilers and build Python!

--
Oscar
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] warning about potential problem for wheels

[Paul Moore]

You could, I suppose, patch your custom Python to *not* treat wheels
built for cp35-none-win32 (or cp35-none-win_amd64, as appropriate) as
compatible with that build (that requires a patch to the tag matching
code in pip). You then wouldn't be able to use any binary wheels on
PyPI, which is a bit heavy handed but probably necessary.

But that's probably way more complex than is justified...
Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] warning about potential problem for wheels

[Laura Creighton]

In a message of Sat, 10 Oct 2015 21:52:58 -, Oscar Benjamin writes:

>Really this is just a case of an unsupported platform. It's unfortunate
>that CPython doesn't properly support this hardware but I think it's
>reasonable that if you have to build your interpreter from source then you
>have to build your extension modules as well.

Alas that there is no easy way to detect.  The situation I am
imagining is where the administrators of a school build pythons for
the students to run on their obsolete hardware, and then the poor
students don't understand why pip doesn't work.  But I suppose we
will just get to deal with that problem when and if it happens.

Laura

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] warning about potential problem for wheels

[Oscar Benjamin]

On Sat, 10 Oct 2015 20:53 Laura Creighton  wrote:

(note, I currently don't have mail delivery on for distutils.  I could
change this, but right now I don't think I have a lot to contribute.
This is just a warning).

If you have old windows hardware, which does not support SSE2, and
windows 7, you can build your own python 3.5.  This will work. But
wheels won't, you will need to build them from source as well.
see: http:// bugs.python.org
/issue25361


This means that wheels could start failing.  It would be good if
the wheels themselves could detect this problem and protest in a
reasonable fashion, but I have no idea if this is possible.  In any
case, I thought you needed to know.



There is no way for wheels to do this. A wheel is just a zip file with a
standardised layout. Pip just extracts the zip, reads the metadata and
copies the files to the appropriate locations. The metadata has no way to
describe the fact that it the wheel contains SSE2 dependent binaries. The
standard tools used to create wheels don't know anything about the contents
of the compiled binaries so they don't really have a way to detect that the
wheel depends on SSE2.

Really this is just a case of an unsupported platform. It's unfortunate
that CPython doesn't properly support this hardware but I think it's
reasonable that if you have to build your interpreter from source then you
have to build your extension modules as well.

I'm not sure of a robust solution to detecting the problem at install time.
Extension module authors can only really guarantee that their Windows
binaries are compatible with standard released binaries. So if someone
builds their own interpreter using different compiler options then there's
no real way for pip or the extension module author to know if the binaries
will be compatible. So either pip rejects all binaries for a non standard
interpreter build or it installs them and hopes for the best.

--
Oscar
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig