Re: [Numpy-discussion] advanced indexing bug with huge arrays?

On Thu, Jan 26, 2012 at 17:39, Sturla Molden wrote: > Den 24.01.2012 17:19, skrev David Warde-Farley: >> >> Hmm. Seeing as the width of a C long is inconsistent, does this imply that >> the random number generator will produce different results on different >> platforms? > > If it does, it is a C

Re: [Numpy-discussion] advanced indexing bug with huge arrays?

Den 24.01.2012 17:19, skrev David Warde-Farley: > > Hmm. Seeing as the width of a C long is inconsistent, does this imply that > the random number generator will produce different results on different > platforms? If it does, it is a C programming mistake. C code should never depend on the exact

Re: [Numpy-discussion] advanced indexing bug with huge arrays?

On Tue, Jan 24, 2012 at 3:30 PM, David Warde-Farley < warde...@iro.umontreal.ca> wrote: > On Tue, Jan 24, 2012 at 01:02:44PM -0500, David Warde-Farley wrote: > > On Tue, Jan 24, 2012 at 06:37:12PM +0100, Robin wrote: > > > > > Yes - I get exactly the same numbers in 64 bit windows with 1.6.1. > >

Re: [Numpy-discussion] advanced indexing bug with huge arrays?

On 24.01.2012 23:30, David Warde-Farley wrote: > I've figured it out. In numpy/core/src/multiarray/mapping.c, PyArray_GetMap > is using an int for a counter variable where it should be using an npy_intp. > > I've filed a pull request at https://github.com/numpy/numpy/pull/188 with a > regression t

Re: [Numpy-discussion] advanced indexing bug with huge arrays?

On 23.01.2012, at 11:23, David Warde-Farley wrote: >> a = numpy.array(numpy.random.randint(256,size=(500,972)),dtype='uint8') >> b = numpy.random.randint(500,size=(4993210,)) >> c = a[b] >> In [14]: c[100:].sum() >> Out[14]: 0 Same here. Python 2.7.2, 64bit, Mac OS X (Lion), 8GB RAM,

Re: [Numpy-discussion] advanced indexing bug with huge arrays?

On Tue, Jan 24, 2012 at 01:02:44PM -0500, David Warde-Farley wrote: > On Tue, Jan 24, 2012 at 06:37:12PM +0100, Robin wrote: > > > Yes - I get exactly the same numbers in 64 bit windows with 1.6.1. > > Alright, so that rules out platform specific effects. > > I'll try and hunt the bug down when

Re: [Numpy-discussion] advanced indexing bug with huge arrays?

On Tue, Jan 24, 2012 at 06:37:12PM +0100, Robin wrote: > Yes - I get exactly the same numbers in 64 bit windows with 1.6.1. Alright, so that rules out platform specific effects. I'll try and hunt the bug down when I have some time, if someone more familiar with the indexing code doesn't beat me

Re: [Numpy-discussion] advanced indexing bug with huge arrays?

On Tue, Jan 24, 2012 at 6:24 PM, David Warde-Farley wrote: > On Tue, Jan 24, 2012 at 06:00:05AM +0100, Sturla Molden wrote: >> Den 23.01.2012 22:08, skrev Christoph Gohlke: >> > >> > Maybe this explains the win-amd64 behavior: There are a couple of places >> > in mtrand where array indices and siz

Re: [Numpy-discussion] advanced indexing bug with huge arrays?

On Tue, Jan 24, 2012 at 06:00:05AM +0100, Sturla Molden wrote: > Den 23.01.2012 22:08, skrev Christoph Gohlke: > > > > Maybe this explains the win-amd64 behavior: There are a couple of places > > in mtrand where array indices and sizes are C long instead of npy_intp, > > for example in the randint

Re: [Numpy-discussion] advanced indexing bug with huge arrays?

On Tue, Jan 24, 2012 at 09:15:01AM +, Robert Kern wrote: > On Tue, Jan 24, 2012 at 08:37, Sturla Molden wrote: > > On 24.01.2012 09:21, Sturla Molden wrote: > > > >> randomkit.c handles C long correctly, I think. There are different codes > >> for 32 and 64 bit C long, and buffer sizes are siz

Re: [Numpy-discussion] advanced indexing bug with huge arrays?

On 24.01.2012 10:15, Robert Kern wrote: > There are two different uses of long that you need to distinguish. One > is for sizes, and one is for parameters and values. The sizes should > definitely be upgraded to npy_intp. The latter shouldn't; these should > remain as the default integer type of P

Re: [Numpy-discussion] advanced indexing bug with huge arrays?

On Tue, Jan 24, 2012 at 09:19, Sturla Molden wrote: > On 24.01.2012 10:16, Robert Kern wrote: > >> I'm sorry, what are you demonstrating there? > > Both npy_intp and C long are used for sizes and indexing. Ah, yes. I think Travis added the multiiter code to cont1_array(), which does broadcasting,

Re: [Numpy-discussion] advanced indexing bug with huge arrays?

On 24.01.2012 10:16, Robert Kern wrote: > I'm sorry, what are you demonstrating there? Both npy_intp and C long are used for sizes and indexing. Sturla ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/

Re: [Numpy-discussion] advanced indexing bug with huge arrays?

On Tue, Jan 24, 2012 at 08:47, Sturla Molden wrote: > The coding is also inconsistent, compare for example: > > https://github.com/numpy/numpy/blob/master/numpy/random/mtrand/mtrand.pyx#L180 > > https://github.com/numpy/numpy/blob/master/numpy/random/mtrand/mtrand.pyx#L201 I'm sorry, what are yo

Re: [Numpy-discussion] advanced indexing bug with huge arrays?

On Tue, Jan 24, 2012 at 08:37, Sturla Molden wrote: > On 24.01.2012 09:21, Sturla Molden wrote: > >> randomkit.c handles C long correctly, I think. There are different codes >> for 32 and 64 bit C long, and buffer sizes are size_t. > > distributions.c take C longs as parameters e.g. for the binomi

Re: [Numpy-discussion] advanced indexing bug with huge arrays?

On 24.01.2012 06:32, Sturla Molden wrote: > Den 24.01.2012 06:00, skrev Sturla Molden: >> Both i and length could overflow here. It should overflow on >> allocation of more than 2 GB. There is also a lot of C longs in the >> internal state (line 55-105), as well as the other functions. > > The use

Re: [Numpy-discussion] advanced indexing bug with huge arrays?

On 24.01.2012 09:21, Sturla Molden wrote: > randomkit.c handles C long correctly, I think. There are different codes > for 32 and 64 bit C long, and buffer sizes are size_t. distributions.c take C longs as parameters e.g. for the binomial distribution. mtrand.pyx correctly handles this, but it c

Re: [Numpy-discussion] advanced indexing bug with huge arrays?

On 24.01.2012 06:32, Sturla Molden wrote: > The use of C long affects all the C and Pyrex source code in mtrand > module, not just mtrand.pyx. All of it is fubar on Win64. randomkit.c handles C long correctly, I think. There are different codes for 32 and 64 bit C long, and buffer sizes are size

Re: [Numpy-discussion] advanced indexing bug with huge arrays?

Den 24.01.2012 06:00, skrev Sturla Molden: > Both i and length could overflow here. It should overflow on > allocation of more than 2 GB. There is also a lot of C longs in the > internal state (line 55-105), as well as the other functions. The use of C long affects all the C and Pyrex source cod

Re: [Numpy-discussion] advanced indexing bug with huge arrays?

Den 23.01.2012 22:08, skrev Christoph Gohlke: > > Maybe this explains the win-amd64 behavior: There are a couple of places > in mtrand where array indices and sizes are C long instead of npy_intp, > for example in the randint function: > >

Re: [Numpy-discussion] advanced indexing bug with huge arrays?

Den 23.01.2012 22:08, skrev Christoph Gohlke: > Maybe this explains the win-amd64 behavior: There are a couple of places > in mtrand where array indices and sizes are C long instead of npy_intp, > for example in the randint function: > >

Re: [Numpy-discussion] advanced indexing bug with huge arrays?

On 1/23/2012 12:33 PM, David Warde-Farley wrote: > On Mon, Jan 23, 2012 at 08:38:44PM +0100, Robin wrote: >> On Mon, Jan 23, 2012 at 7:55 PM, David Warde-Farley >> wrote: >>> I've reproduced this (rather serious) bug myself and confirmed that it >>> exists >>> in master, and as far back as 1.4

Re: [Numpy-discussion] advanced indexing bug with huge arrays?

On Mon, Jan 23, 2012 at 08:38:44PM +0100, Robin wrote: > On Mon, Jan 23, 2012 at 7:55 PM, David Warde-Farley > wrote: > > I've reproduced this (rather serious) bug myself and confirmed that it > > exists > > in master, and as far back as 1.4.1. > > > > I'd really appreciate if someone could repro

Re: [Numpy-discussion] advanced indexing bug with huge arrays?

Hi Travis, Thanks for your reply. On Mon, Jan 23, 2012 at 01:33:42PM -0600, Travis Oliphant wrote: > Can you determine where the problem is, precisely.In other words, can you > verify that c is not getting filled in correctly? > > You are no doubt going to get overflow in the summation as

Re: [Numpy-discussion] advanced indexing bug with huge arrays?

On Mon, Jan 23, 2012 at 1:33 PM, Travis Oliphant wrote: > Can you determine where the problem is, precisely.In other words, can > you verify that c is not getting filled in correctly? > > You are no doubt going to get overflow in the summation as you have a > uint8 parameter. But, having tha

Re: [Numpy-discussion] advanced indexing bug with huge arrays?

On Mon, Jan 23, 2012 at 7:55 PM, David Warde-Farley wrote: > I've reproduced this (rather serious) bug myself and confirmed that it exists > in master, and as far back as 1.4.1. > > I'd really appreciate if someone could reproduce and confirm on another > machine, as so far all my testing has been

Re: [Numpy-discussion] advanced indexing bug with huge arrays?

Can you determine where the problem is, precisely.In other words, can you verify that c is not getting filled in correctly? You are no doubt going to get overflow in the summation as you have a uint8 parameter. But, having that overflow be exactly '0' would be surprising. Can you verif

Re: [Numpy-discussion] advanced indexing bug with huge arrays?

I've reproduced this (rather serious) bug myself and confirmed that it exists in master, and as far back as 1.4.1. I'd really appreciate if someone could reproduce and confirm on another machine, as so far all my testing has been on our single high-memory machine. Thanks, David On Mon, Jan 23, 2

[Numpy-discussion] advanced indexing bug with huge arrays?

A colleague has run into this weird behaviour with NumPy 1.6.1, EPD 7.1-2, on Linux (Fedora Core 14) 64-bit: > a = numpy.array(numpy.random.randint(256,size=(500,972)),dtype='uint8') > b = numpy.random.randint(500,size=(4993210,)) > c = a[b] It seems c is not getting filled in full, name