Re: [Numpy-discussion] change default integer from int32 to int64 on win64?
On Sat, Jul 26, 2014 at 9:19 AM, Lars Buitinck wrote:
>> Date: Fri, 25 Jul 2014 15:06:40 +0200
>> From: Olivier Grisel
>> Subject: Re: [Numpy-discussion] change default integer from int32 to
>> int64 on win64?
>> To: Discussion of Numerical Python
>> Content-Type: text/plain; charset=UTF-8
>>
>> The dtype returned by np.where looks right (int64):
>>
>>>>> import platform
>>>>> platform.architecture()
>> ('64bit', 'WindowsPE')
>>>>> import numpy as np
>>>>> np.__version__
>> '1.9.0b1'
>>>>> a = np.zeros(10)
>>>>> np.where(a == 0)
>> (array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=int64),)
>
> Strange. In [1] we had to cast the result of np.where because it was
> an array of long. I ran through the NumPy code, and I couldn't find
> the flaw, but neither could I find a point in the history where it was
> fixed.
>
> [1]
> https://github.com/scikit-learn/scikit-learn/commit/ebdeddbab1620c2473d04dc242d1e30684af9511
As far as I can tell, it's been that way essentially forever, before
numpy was numpy:
https://github.com/numpy/numpy/commit/8cb36a62#diff-88aedadb94e0ead6b434d55f81668471R645
--
Robert Kern
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] change default integer from int32 to int64 on win64?
> Date: Fri, 25 Jul 2014 15:06:40 +0200
> From: Olivier Grisel
> Subject: Re: [Numpy-discussion] change default integer from int32 to
> int64 on win64?
> To: Discussion of Numerical Python
> Content-Type: text/plain; charset=UTF-8
>
> The dtype returned by np.where looks right (int64):
>
>>>> import platform
>>>> platform.architecture()
> ('64bit', 'WindowsPE')
>>>> import numpy as np
>>>> np.__version__
> '1.9.0b1'
>>>> a = np.zeros(10)
>>>> np.where(a == 0)
> (array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=int64),)
Strange. In [1] we had to cast the result of np.where because it was
an array of long. I ran through the NumPy code, and I couldn't find
the flaw, but neither could I find a point in the history where it was
fixed.
[1]
https://github.com/scikit-learn/scikit-learn/commit/ebdeddbab1620c2473d04dc242d1e30684af9511
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] change default integer from int32 to int64 on win64?
The dtype returned by np.where looks right (int64):
>>> import platform
>>> platform.architecture()
('64bit', 'WindowsPE')
>>> import numpy as np
>>> np.__version__
'1.9.0b1'
>>> a = np.zeros(10)
>>> np.where(a == 0)
(array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=int64),)
--
Olivier
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] change default integer from int32 to int64 on win64?
On Thu, Jul 24, 2014 at 10:39 AM, Lars Buitinck wrote: > Wed, 23 Jul 2014 22:13:33 +0100 Nathaniel Smith : >> On Wed, Jul 23, 2014 at 9:57 PM, Robert Kern wrote: >>> That's perhaps what you want, but numpy has never claimed to do this. > > ... except in np.where, which promises to return indices but actually > returns arrays of longs and thus doesn't work with large arrays on > Windows. > > I know this is a bug that can be fixed without changing the size of > np.int, but it goes to show that even core functionality in NumPy gets > it wrong. Does it? I don't have my Windows VM available at the moment, but it looks like PyArray_Nonzero() is correctly returning an intp array: https://github.com/numpy/numpy/blob/master/numpy/core/src/multiarray/item_selection.c#L2478 If it is incorrect somewhere else, please submit a bug report. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] change default integer from int32 to int64 on win64?
Wed, 23 Jul 2014 22:13:33 +0100 Nathaniel Smith : > On Wed, Jul 23, 2014 at 9:57 PM, Robert Kern wrote: >> That's perhaps what you want, but numpy has never claimed to do this. ... except in np.where, which promises to return indices but actually returns arrays of longs and thus doesn't work with large arrays on Windows. I know this is a bug that can be fixed without changing the size of np.int, but it goes to show that even core functionality in NumPy gets it wrong. > This is true, but it's not very compelling on its own -- "big as a > pointer" is a much much more useful property than "big as a long". The > only real reason this made sense in the first place is the equivalence > between Python int and C long, but even that is gone now with Python > 3. IMO at this point backcompat is really the only serious reason for > keeping int32 as the default integer type in win64. But of course this > is a pretty serious concern... Hear, hear. The C type long is only useful as an "at least 32-bit" integer, but on the platforms that NumPy targets, int is also at least that large. The only real benefit of long is that it makes porting more interesting . If you have intp and a bunch of explicitly-sized integer types, you don't need an additional type that behaves like a long *except* for backward compat. The Go people got this right; they only have explicitly-sized integer types and an int type the size of a pointer [1]. [1] http://golang.org/doc/go1.1#int ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] change default integer from int32 to int64 on win64?
On Thu, Jul 24, 2014 at 3:47 AM, Sturla Molden wrote: > Julian Taylor wrote: > >> The default integer dtype should be sufficiently large to index into any >> numpy array, thats what I call an API here. win64 behaves different, you >> have to explicitly upcast your index to be able to index all memory. > > No, you don't have to manually upcast Python int to Python long. > > Python 2 will automatically create a Python long if you overflow a Python > int. > > On Python 3 the Python int does not have a size limit. Please reread the thread more carefully. That's not what this discussion is about. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] change default integer from int32 to int64 on win64?
Julian Taylor wrote: > The default integer dtype should be sufficiently large to index into any > numpy array, thats what I call an API here. win64 behaves different, you > have to explicitly upcast your index to be able to index all memory. No, you don't have to manually upcast Python int to Python long. Python 2 will automatically create a Python long if you overflow a Python int. On Python 3 the Python int does not have a size limit. Sturla ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] change default integer from int32 to int64 on win64?
On Wed, Jul 23, 2014 at 9:57 PM, Robert Kern wrote: > That's perhaps what you want, but numpy has never claimed to do this. > The numpy project deliberately chose (and is so documented) to make > its default integer type a C long, not a C size_t, to match Python's > default. This is true, but it's not very compelling on its own -- "big as a pointer" is a much much more useful property than "big as a long". The only real reason this made sense in the first place is the equivalence between Python int and C long, but even that is gone now with Python 3. IMO at this point backcompat is really the only serious reason for keeping int32 as the default integer type in win64. But of course this is a pretty serious concern... Julian: making the change experimentally and checking how badly scipy and some similar libraries break might be a way to focus the backcompat discussion more. -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] change default integer from int32 to int64 on win64?
On Wed, Jul 23, 2014 at 9:57 PM, Robert Kern wrote:
> That's what I'm suggesting that we change: make
> `type(ndarray.shape[i])` be `np.intp` instead of `long`.
>
> However, I'm not sure that this is an issue with numpy 1.8.0 at least.
> I can't reproduce the reported problem on Win64:
>
> In [12]: import numpy as np
>
> In [13]: from numpy.lib import stride_tricks
>
> In [14]: import sys
>
> In [15]: b = stride_tricks.as_strided(np.zeros(1), shape=(10,
> 20, 40), strides=(0, 0, 0))
>
> In [16]: b.shape
> Out[16]: (10L, 20L, 40L)
>
> In [17]: np.product(b.shape)
> Out[17]: 8000
>
> In [18]: np.product(b.shape).dtype
> Out[18]: dtype('int64')
>
> In [19]: sys.maxint
> Out[19]: 2147483647
>
> In [20]: np.__version__
> Out[20]: '1.8.0'
>
> In [21]: np.array(b.shape)
> Out[21]: array([10, 20, 40], dtype=int64)
>
>
> This is on Python 2.7, so maybe something got weird in the Python 3
> version that Chris Gohlke tested?
Ah yes, naturally. Because there is no separate `long` type in Python
3, np.asarray() can't use the type to distinguish what type to build
the array. Returning np.intp objects in the tuple would resolve the
problem in much the same way the problem is currently resolved in
Python 2. This would also have the effect of unifying API on all
platforms: currently, win64 is the only platform where the `.shape`
tuple and related attribute returns Python longs instead of Python
ints.
--
Robert Kern
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] change default integer from int32 to int64 on win64?
On Wed, Jul 23, 2014 at 9:34 PM, Julian Taylor
wrote:
> On 23.07.2014 22:04, Robert Kern wrote:
>> On Wed, Jul 23, 2014 at 8:50 PM, Julian Taylor
>> wrote:
>>> On 23.07.2014 20:54, Robert Kern wrote:
On Wed, Jul 23, 2014 at 6:19 PM, Julian Taylor
wrote:
> hi,
> it recently came to my attention that the default integer type in numpy
> on windows 64 bit is a 32 bit integers [0].
> This seems like a quite serious problem as it means you can't use any
> integers created from python integers < 32 bit to index arrays larger
> than 2GB.
> For example np.product(array.shape) which will never overflow on linux
> and mac, can overflow on win64.
Currently, on win64, we use Python long integer objects for `.shape`
and related attributes. I wonder if we could return numpy int64
scalars instead. Then np.product() (or anything else that consumes
these via np.asarray()) would infer the correct dtype for the result.
>>>
>>> this might be a less invasive alternative that might solve a lot of the
>>> incompatibilities, but it would probably also change np.arange(5) and
>>> similar functions to int64 which might change the dtype of a lot of
>>> arrays. The difference to just changing it everywhere might not be so
>>> large anymore.
>>
>> No, np.arange(5) would not change behavior given my suggestion, only
>> the type of the integer objects in ndarray.shape and related tuples.
>
> ndarray.shape are not numpy scalars but python objects, so they would
> always be converted back to 32 bit integers when given back to numpy.
That's what I'm suggesting that we change: make
`type(ndarray.shape[i])` be `np.intp` instead of `long`.
However, I'm not sure that this is an issue with numpy 1.8.0 at least.
I can't reproduce the reported problem on Win64:
In [12]: import numpy as np
In [13]: from numpy.lib import stride_tricks
In [14]: import sys
In [15]: b = stride_tricks.as_strided(np.zeros(1), shape=(10,
20, 40), strides=(0, 0, 0))
In [16]: b.shape
Out[16]: (10L, 20L, 40L)
In [17]: np.product(b.shape)
Out[17]: 8000
In [18]: np.product(b.shape).dtype
Out[18]: dtype('int64')
In [19]: sys.maxint
Out[19]: 2147483647
In [20]: np.__version__
Out[20]: '1.8.0'
In [21]: np.array(b.shape)
Out[21]: array([10, 20, 40], dtype=int64)
This is on Python 2.7, so maybe something got weird in the Python 3
version that Chris Gohlke tested?
> I think this is a very dangerous platform difference and a quite large
> inconvenience for win64 users so I think it would be good to fix this.
> This would be a very large change of API and probably also ABI.
Yes. Not only would it be a very large change from the status quo, I
think it introduces *much greater* platform difference than what we
have currently. The assumption that the default integer object
corresponds to the platform C long, whatever that is, is pretty
heavily ingrained.
>>>
>>> This should be only a concern for the ABI which can be solved by simply
>>> recompiling.
>>> In comparison that the API is different on win64 compared to all other
>>> platforms is something that needs source level changes.
>>
>> No, the API is no different on win64 than other platforms. Why do you
>> think it is? The win64 platform is a weird platform in this respect,
>> having made a choice that other 64-bit platforms didn't, but numpy's
>> API treats it consistently. When we say that something is a C long,
>> it's a C long on all platforms.
>
> The API is different if you consider it from a python perspective.
> The default integer dtype should be sufficiently large to index into any
> numpy array, thats what I call an API here.
That's perhaps what you want, but numpy has never claimed to do this.
The numpy project deliberately chose (and is so documented) to make
its default integer type a C long, not a C size_t, to match Python's
default.
> win64 behaves different, you
> have to explicitly upcast your index to be able to index all memory.
> But API or ABI is just semantics here, what I actually mean is the
> difference of source changes vs recompiling to deal with the issue.
> Of course there might be C code that needs more than recompiling, but it
> should not be that much, it would have to be already somewhat
> broken/restrictive code that uses numpy buffers without first checking
> which type it has.
>
> There can also be python code that might need source changes e.g.
> np.int_ memory mapping a binary from win32 assuming np.int_ is also 32
> bit on win64, but this would be broken on linux and mac already now.
Anything that assumes that np.int_ is any particular fixed size is
always broken, naturally.
--
Robert Kern
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] change default integer from int32 to int64 on win64?
On 23.07.2014 22:04, Robert Kern wrote: > On Wed, Jul 23, 2014 at 8:50 PM, Julian Taylor > wrote: >> On 23.07.2014 20:54, Robert Kern wrote: >>> On Wed, Jul 23, 2014 at 6:19 PM, Julian Taylor >>> wrote: hi, it recently came to my attention that the default integer type in numpy on windows 64 bit is a 32 bit integers [0]. This seems like a quite serious problem as it means you can't use any integers created from python integers < 32 bit to index arrays larger than 2GB. For example np.product(array.shape) which will never overflow on linux and mac, can overflow on win64. >>> >>> Currently, on win64, we use Python long integer objects for `.shape` >>> and related attributes. I wonder if we could return numpy int64 >>> scalars instead. Then np.product() (or anything else that consumes >>> these via np.asarray()) would infer the correct dtype for the result. >> >> this might be a less invasive alternative that might solve a lot of the >> incompatibilities, but it would probably also change np.arange(5) and >> similar functions to int64 which might change the dtype of a lot of >> arrays. The difference to just changing it everywhere might not be so >> large anymore. > > No, np.arange(5) would not change behavior given my suggestion, only > the type of the integer objects in ndarray.shape and related tuples. ndarray.shape are not numpy scalars but python objects, so they would always be converted back to 32 bit integers when given back to numpy. > I think this is a very dangerous platform difference and a quite large inconvenience for win64 users so I think it would be good to fix this. This would be a very large change of API and probably also ABI. >>> >>> Yes. Not only would it be a very large change from the status quo, I >>> think it introduces *much greater* platform difference than what we >>> have currently. The assumption that the default integer object >>> corresponds to the platform C long, whatever that is, is pretty >>> heavily ingrained. >> >> This should be only a concern for the ABI which can be solved by simply >> recompiling. >> In comparison that the API is different on win64 compared to all other >> platforms is something that needs source level changes. > > No, the API is no different on win64 than other platforms. Why do you > think it is? The win64 platform is a weird platform in this respect, > having made a choice that other 64-bit platforms didn't, but numpy's > API treats it consistently. When we say that something is a C long, > it's a C long on all platforms. The API is different if you consider it from a python perspective. The default integer dtype should be sufficiently large to index into any numpy array, thats what I call an API here. win64 behaves different, you have to explicitly upcast your index to be able to index all memory. But API or ABI is just semantics here, what I actually mean is the difference of source changes vs recompiling to deal with the issue. Of course there might be C code that needs more than recompiling, but it should not be that much, it would have to be already somewhat broken/restrictive code that uses numpy buffers without first checking which type it has. There can also be python code that might need source changes e.g. np.int_ memory mapping a binary from win32 assuming np.int_ is also 32 bit on win64, but this would be broken on linux and mac already now. But as we also never officially released win64 binaries we could change it for from source compilations and give win64 binary distributors the option to keep the old ABI/API at their discretion. >>> >>> That option would make the problem worse, not better. >> >> maybe, I'm not familiar with the numpy win64 distribution landscape. >> Is it not like linux where you have one distributor per workstation >> setup that can update all its packages to a new ABI on one go? > > No. There tend to be multiple providers. > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] change default integer from int32 to int64 on win64?
On Wed, 2014-07-23 at 22:06 +0200, Sebastian Berg wrote: > On Wed, 2014-07-23 at 21:50 +0200, Julian Taylor wrote: > > On 23.07.2014 20:54, Robert Kern wrote: > > > On Wed, Jul 23, 2014 at 6:19 PM, Julian Taylor > > > wrote: > > >> hi, > > >> it recently came to my attention that the default integer type in numpy > > >> on windows 64 bit is a 32 bit integers [0]. > > >> This seems like a quite serious problem as it means you can't use any > > >> integers created from python integers < 32 bit to index arrays larger > > >> than 2GB. > > >> For example np.product(array.shape) which will never overflow on linux > > >> and mac, can overflow on win64. > > > > > > Currently, on win64, we use Python long integer objects for `.shape` > > > and related attributes. I wonder if we could return numpy int64 > > > scalars instead. Then np.product() (or anything else that consumes > > > these via np.asarray()) would infer the correct dtype for the result. > > > > this might be a less invasive alternative that might solve a lot of the > > incompatibilities, but it would probably also change np.arange(5) and > > similar functions to int64 which might change the dtype of a lot of > > arrays. The difference to just changing it everywhere might not be so > > large anymore. > > > > Aren't most such functions already using intp? Just guessing, but: > > In [16]: np.arange(30, dtype=np.long).dtype.num > Out[16]: 9 > > In [17]: np.arange(30, dtype=np.intp).dtype.num > Out[17]: 7 > > In [18]: np.arange(30).dtype.num > Out[18]: 7 > Ops, never mind that stuff, probably not... np.int_ is 7 too, this is just the way how intp is chosen. > frankly, I am not sure what needs to change at all, except the normal > array creation and the sum promotion rule. I am probably naive here, but > what is the ABI change that is necessary for that? > > I guess the problem you see is breaking code doing np.array([1,2,3]) and > then assuming in C that it is a long array? > > - Sebastian > > > > > > >> I think this is a very dangerous platform difference and a quite large > > >> inconvenience for win64 users so I think it would be good to fix this. > > >> This would be a very large change of API and probably also ABI. > > > > > > Yes. Not only would it be a very large change from the status quo, I > > > think it introduces *much greater* platform difference than what we > > > have currently. The assumption that the default integer object > > > corresponds to the platform C long, whatever that is, is pretty > > > heavily ingrained. > > > > This should be only a concern for the ABI which can be solved by simply > > recompiling. > > In comparison that the API is different on win64 compared to all other > > platforms is something that needs source level changes. > > > > > > > >> But as we also never officially released win64 binaries we could change > > >> it for from source compilations and give win64 binary distributors the > > >> option to keep the old ABI/API at their discretion. > > > > > > That option would make the problem worse, not better. > > > > > > > maybe, I'm not familiar with the numpy win64 distribution landscape. > > Is it not like linux where you have one distributor per workstation > > setup that can update all its packages to a new ABI on one go? > > ___ > > NumPy-Discussion mailing list > > NumPy-Discussion@scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] change default integer from int32 to int64 on win64?
On Wed, 2014-07-23 at 21:50 +0200, Julian Taylor wrote: > On 23.07.2014 20:54, Robert Kern wrote: > > On Wed, Jul 23, 2014 at 6:19 PM, Julian Taylor > > wrote: > >> hi, > >> it recently came to my attention that the default integer type in numpy > >> on windows 64 bit is a 32 bit integers [0]. > >> This seems like a quite serious problem as it means you can't use any > >> integers created from python integers < 32 bit to index arrays larger > >> than 2GB. > >> For example np.product(array.shape) which will never overflow on linux > >> and mac, can overflow on win64. > > > > Currently, on win64, we use Python long integer objects for `.shape` > > and related attributes. I wonder if we could return numpy int64 > > scalars instead. Then np.product() (or anything else that consumes > > these via np.asarray()) would infer the correct dtype for the result. > > this might be a less invasive alternative that might solve a lot of the > incompatibilities, but it would probably also change np.arange(5) and > similar functions to int64 which might change the dtype of a lot of > arrays. The difference to just changing it everywhere might not be so > large anymore. > Aren't most such functions already using intp? Just guessing, but: In [16]: np.arange(30, dtype=np.long).dtype.num Out[16]: 9 In [17]: np.arange(30, dtype=np.intp).dtype.num Out[17]: 7 In [18]: np.arange(30).dtype.num Out[18]: 7 frankly, I am not sure what needs to change at all, except the normal array creation and the sum promotion rule. I am probably naive here, but what is the ABI change that is necessary for that? I guess the problem you see is breaking code doing np.array([1,2,3]) and then assuming in C that it is a long array? - Sebastian > > > >> I think this is a very dangerous platform difference and a quite large > >> inconvenience for win64 users so I think it would be good to fix this. > >> This would be a very large change of API and probably also ABI. > > > > Yes. Not only would it be a very large change from the status quo, I > > think it introduces *much greater* platform difference than what we > > have currently. The assumption that the default integer object > > corresponds to the platform C long, whatever that is, is pretty > > heavily ingrained. > > This should be only a concern for the ABI which can be solved by simply > recompiling. > In comparison that the API is different on win64 compared to all other > platforms is something that needs source level changes. > > > > >> But as we also never officially released win64 binaries we could change > >> it for from source compilations and give win64 binary distributors the > >> option to keep the old ABI/API at their discretion. > > > > That option would make the problem worse, not better. > > > > maybe, I'm not familiar with the numpy win64 distribution landscape. > Is it not like linux where you have one distributor per workstation > setup that can update all its packages to a new ABI on one go? > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] change default integer from int32 to int64 on win64?
On Wed, Jul 23, 2014 at 8:50 PM, Julian Taylor wrote: > On 23.07.2014 20:54, Robert Kern wrote: >> On Wed, Jul 23, 2014 at 6:19 PM, Julian Taylor >> wrote: >>> hi, >>> it recently came to my attention that the default integer type in numpy >>> on windows 64 bit is a 32 bit integers [0]. >>> This seems like a quite serious problem as it means you can't use any >>> integers created from python integers < 32 bit to index arrays larger >>> than 2GB. >>> For example np.product(array.shape) which will never overflow on linux >>> and mac, can overflow on win64. >> >> Currently, on win64, we use Python long integer objects for `.shape` >> and related attributes. I wonder if we could return numpy int64 >> scalars instead. Then np.product() (or anything else that consumes >> these via np.asarray()) would infer the correct dtype for the result. > > this might be a less invasive alternative that might solve a lot of the > incompatibilities, but it would probably also change np.arange(5) and > similar functions to int64 which might change the dtype of a lot of > arrays. The difference to just changing it everywhere might not be so > large anymore. No, np.arange(5) would not change behavior given my suggestion, only the type of the integer objects in ndarray.shape and related tuples. >>> I think this is a very dangerous platform difference and a quite large >>> inconvenience for win64 users so I think it would be good to fix this. >>> This would be a very large change of API and probably also ABI. >> >> Yes. Not only would it be a very large change from the status quo, I >> think it introduces *much greater* platform difference than what we >> have currently. The assumption that the default integer object >> corresponds to the platform C long, whatever that is, is pretty >> heavily ingrained. > > This should be only a concern for the ABI which can be solved by simply > recompiling. > In comparison that the API is different on win64 compared to all other > platforms is something that needs source level changes. No, the API is no different on win64 than other platforms. Why do you think it is? The win64 platform is a weird platform in this respect, having made a choice that other 64-bit platforms didn't, but numpy's API treats it consistently. When we say that something is a C long, it's a C long on all platforms. >>> But as we also never officially released win64 binaries we could change >>> it for from source compilations and give win64 binary distributors the >>> option to keep the old ABI/API at their discretion. >> >> That option would make the problem worse, not better. > > maybe, I'm not familiar with the numpy win64 distribution landscape. > Is it not like linux where you have one distributor per workstation > setup that can update all its packages to a new ABI on one go? No. There tend to be multiple providers. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] change default integer from int32 to int64 on win64?
On 23.07.2014 20:54, Robert Kern wrote: > On Wed, Jul 23, 2014 at 6:19 PM, Julian Taylor > wrote: >> hi, >> it recently came to my attention that the default integer type in numpy >> on windows 64 bit is a 32 bit integers [0]. >> This seems like a quite serious problem as it means you can't use any >> integers created from python integers < 32 bit to index arrays larger >> than 2GB. >> For example np.product(array.shape) which will never overflow on linux >> and mac, can overflow on win64. > > Currently, on win64, we use Python long integer objects for `.shape` > and related attributes. I wonder if we could return numpy int64 > scalars instead. Then np.product() (or anything else that consumes > these via np.asarray()) would infer the correct dtype for the result. this might be a less invasive alternative that might solve a lot of the incompatibilities, but it would probably also change np.arange(5) and similar functions to int64 which might change the dtype of a lot of arrays. The difference to just changing it everywhere might not be so large anymore. > >> I think this is a very dangerous platform difference and a quite large >> inconvenience for win64 users so I think it would be good to fix this. >> This would be a very large change of API and probably also ABI. > > Yes. Not only would it be a very large change from the status quo, I > think it introduces *much greater* platform difference than what we > have currently. The assumption that the default integer object > corresponds to the platform C long, whatever that is, is pretty > heavily ingrained. This should be only a concern for the ABI which can be solved by simply recompiling. In comparison that the API is different on win64 compared to all other platforms is something that needs source level changes. > >> But as we also never officially released win64 binaries we could change >> it for from source compilations and give win64 binary distributors the >> option to keep the old ABI/API at their discretion. > > That option would make the problem worse, not better. > maybe, I'm not familiar with the numpy win64 distribution landscape. Is it not like linux where you have one distributor per workstation setup that can update all its packages to a new ABI on one go? ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] change default integer from int32 to int64 on win64?
On Wed, Jul 23, 2014 at 6:19 PM, Julian Taylor wrote: > hi, > it recently came to my attention that the default integer type in numpy > on windows 64 bit is a 32 bit integers [0]. > This seems like a quite serious problem as it means you can't use any > integers created from python integers < 32 bit to index arrays larger > than 2GB. > For example np.product(array.shape) which will never overflow on linux > and mac, can overflow on win64. Currently, on win64, we use Python long integer objects for `.shape` and related attributes. I wonder if we could return numpy int64 scalars instead. Then np.product() (or anything else that consumes these via np.asarray()) would infer the correct dtype for the result. > I think this is a very dangerous platform difference and a quite large > inconvenience for win64 users so I think it would be good to fix this. > This would be a very large change of API and probably also ABI. Yes. Not only would it be a very large change from the status quo, I think it introduces *much greater* platform difference than what we have currently. The assumption that the default integer object corresponds to the platform C long, whatever that is, is pretty heavily ingrained. > But as we also never officially released win64 binaries we could change > it for from source compilations and give win64 binary distributors the > option to keep the old ABI/API at their discretion. That option would make the problem worse, not better. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] change default integer from int32 to int64 on win64?
hi, it recently came to my attention that the default integer type in numpy on windows 64 bit is a 32 bit integers [0]. This seems like a quite serious problem as it means you can't use any integers created from python integers < 32 bit to index arrays larger than 2GB. For example np.product(array.shape) which will never overflow on linux and mac, can overflow on win64. I think this is a very dangerous platform difference and a quite large inconvenience for win64 users so I think it would be good to fix this. This would be a very large change of API and probably also ABI. But as we also never officially released win64 binaries we could change it for from source compilations and give win64 binary distributors the option to keep the old ABI/API at their discretion. Any thoughts on this from win64 users? Cheers, Julian Taylor [0] https://github.com/astropy/astropy/pull/2697 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
