Re: [Numpy-discussion] Question about flags of fancy indexed array

[Travis Oliphant]

Charles R Harris wrote:
>
>
> On 5/23/07, *Albert Strasheim* <[EMAIL PROTECTED] 
> > wrote:
>
> Hello all
>
> On Wed, 23 May 2007, Anne Archibald wrote:
>
> > On 23/05/07, Albert Strasheim <[EMAIL PROTECTED]
> > wrote:
> >
> > > Consider the following example:
> >
> > First a comment: almost nobody needs to care how the data is stored
> > internally. Try to avoid looking at the flags unless you're
> > interfacing with a C library. The nice feature of numpy is that it
> > hides all that junk - strides, contiguous storage, iteration, what
> > have you - so that you don't have to deal with it.
>
> As luck would have it, I am interfacing with a C library.
>
> > > Is it correct that the F_CONTIGUOUS flag is set in the case of
> the fancy
> > > indexed x? I'm running NumPy 1.0.3.dev3792 here.
> >
> > Numpy arrays are always stored in contiguous blocks of memory with
> > uniform strides. The "CONTIGUOUS" flag actually means something
> > totally different, which is unfortunate, but in any case, "fancy
> > indexing" can't be done as a simple reindexing operation. It
> must make
> > a copy of the array. So what you're seeing is the flags of a
> fresh new
> > array, created from scratch (and numpy always creates arrays in C
> > order internally, though that is an implementation detail you should
> > not rely on).
>
> If you are correct that this is in fact a fresh new array, I really
> don't understand where the values of these flags. To recap:
>
> In [19]: x = N.zeros((3,2))
>
> In [20]: x.flags
> Out[20]:
>   C_CONTIGUOUS : True
>   F_CONTIGUOUS : False
>   OWNDATA : True
>   WRITEABLE : True
>   ALIGNED : True
>   UPDATEIFCOPY : False
>
> In [21]: x[:,[1,0]].flags
> Out[21]:
>   C_CONTIGUOUS : False
>   F_CONTIGUOUS : True
>   OWNDATA : False
>   WRITEABLE : True
>   ALIGNED : True
>   UPDATEIFCOPY : False
>
> So since x and x[:,[1,0]] are both new arrays, shouldn't their
> flags be
> identical? I'd expect at least C_CONTIGUOUS and OWNDATA to be True.
>
>
> The contiguous refers to how stuff is layed out in memory. In this 
> case it appears that fancy indexing creates the new array by first 
> copying column 1, then column 2, so that the new array is indeed 
> F_CONTIGUOUS. Assuming I correctly understand the behaviour of the 
> tostring argument, which is debatable, that is indeed what happens.

Yes, you are using the 'A' argument correctly.  For contiguous arrays, 
you can always get a look at the actual buffer, using

x.data[:]
or
y.data[:]

You can't get a buffer for discontiguous arrays.

-Travis

___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Question about flags of fancy indexed array

[Travis Oliphant]

Albert Strasheim wrote:
> Hello all
>
> Consider the following example:
>
> In [43]: x = N.zeros((3,2))
>
> In [44]: x.flags
> Out[44]:
>   C_CONTIGUOUS : True
>   F_CONTIGUOUS : False
>   OWNDATA : True
>   WRITEABLE : True
>   ALIGNED : True
>   UPDATEIFCOPY : False
>
> In [45]: x[:,[1,0]].flags
> Out[45]:
>   C_CONTIGUOUS : False
>   F_CONTIGUOUS : True
>   OWNDATA : False
>   WRITEABLE : True
>   ALIGNED : True
>   UPDATEIFCOPY : False
>
> Is it correct that the F_CONTIGUOUS flag is set in the case of the fancy 
> indexed x? I'm running NumPy 1.0.3.dev3792 here.
>   

In this case, yes.  When you use fancy-indexing with standard slicing 
(an extension that NumPy added), the implementation uses transposes 
under the covers quite often.   So, you can't rely on the output of 
fancy indexing being a C-contiguous array (even though it won't be 
referring to the same data as the original array). 

So, you are exposing an implementation detail here.  To interface to 
code that requires C-contiguous or F-contiguous data, you have to check 
the flags and make an appropriate copy.

-Travis

___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Question about flags of fancy indexed array

[Timothy Hochberg]

I'm not sure why the data ends up F_CONTIGUOUS, but it appears that you can
get things to end up as C_CONTIGUOUS by transposing before the indexing and
then transposing back. I don't think that this results in extra copies, but
I'm not certain of that.


a = np.arange(6).reshape(3,2)
a

array([[0, 1],
  [2, 3],
  [4, 5]])

b1 = a[:,[1,0]]
a.flags

 C_CONTIGUOUS : True
 F_CONTIGUOUS : False
 OWNDATA : False
 WRITEABLE : True
 ALIGNED : True
 UPDATEIFCOPY : False

b1.flags

 C_CONTIGUOUS : False
 F_CONTIGUOUS : True
 OWNDATA : False
 WRITEABLE : True
 ALIGNED : True
 UPDATEIFCOPY : False

b2 = a.transpose()[[(1,0)]].transpose()
b2.flags

 C_CONTIGUOUS : True
 F_CONTIGUOUS : False
 OWNDATA : False
 WRITEABLE : True
 ALIGNED : True
 UPDATEIFCOPY : False

b1 == b2

array([[True, True],
  [True, True],
  [True, True]], dtype=bool)


--

//=][=\\

[EMAIL PROTECTED]
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Question about flags of fancy indexed array

[Charles R Harris]

On 5/23/07, Anne Archibald <[EMAIL PROTECTED]> wrote:


On 23/05/07, Albert Strasheim <[EMAIL PROTECTED]> wrote:

> If you are correct that this is in fact a fresh new array, I really
> don't understand where the values of these flags. To recap:
>
> In [19]: x = N.zeros((3,2))
>
> In [20]: x.flags
> Out[20]:
>   C_CONTIGUOUS : True
>   F_CONTIGUOUS : False
>   OWNDATA : True
>   WRITEABLE : True
>   ALIGNED : True
>   UPDATEIFCOPY : False
>
> In [21]: x[:,[1,0]].flags
> Out[21]:
>   C_CONTIGUOUS : False
>   F_CONTIGUOUS : True
>   OWNDATA : False
>   WRITEABLE : True
>   ALIGNED : True
>   UPDATEIFCOPY : False
>
> So since x and x[:,[1,0]] are both new arrays, shouldn't their flags be
> identical? I'd expect at least C_CONTIGUOUS and OWNDATA to be True.

It looks like x[:,[1,0]] is done by fancy indexing on the first index
and then transposing. I haven't looked at the implementation, though.
If you need the result to be C-contiguous without further copying, you
can do:
x.transpose()[[1,0],:].transpose().flags
which is horrible. I wouldn't rely on it, though, I'd use
asconguousarray afterward.



Both copy and ascontiguousarray do the job:

In [39]: ascontiguousarray(x[:,[1,0]]).flags
Out[39]:
 C_CONTIGUOUS : True
 F_CONTIGUOUS : False
 OWNDATA : True
 WRITEABLE : True
 ALIGNED : True
 UPDATEIFCOPY : False

In [40]: x[:,[1,0]].copy().flags
Out[40]:
 C_CONTIGUOUS : True
 F_CONTIGUOUS : False
 OWNDATA : True
 WRITEABLE : True
 ALIGNED : True
 UPDATEIFCOPY : False

The extra copy is unfortunate but I don't see how it is to be avoided.
Maintaining order and contiguity in memory is not the goal of numpy,
convenience and flexibility is. So interfacing with external C and FORTRAN
is always going to need some care.

Chuck
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Question about flags of fancy indexed array

[Charles R Harris]

On 5/23/07, Charles R Harris <[EMAIL PROTECTED]> wrote:




On 5/23/07, Albert Strasheim <[EMAIL PROTECTED]> wrote:
>
> Hello all
>
> On Wed, 23 May 2007, Anne Archibald wrote:
>
> > On 23/05/07, Albert Strasheim <[EMAIL PROTECTED]> wrote:
> >
> > > Consider the following example:
> >
> > First a comment: almost nobody needs to care how the data is stored
> > internally. Try to avoid looking at the flags unless you're
> > interfacing with a C library. The nice feature of numpy is that it
> > hides all that junk - strides, contiguous storage, iteration, what
> > have you - so that you don't have to deal with it.
>
> As luck would have it, I am interfacing with a C library.
>
> > > Is it correct that the F_CONTIGUOUS flag is set in the case of the
> fancy
> > > indexed x? I'm running NumPy 1.0.3.dev3792 here.
> >
> > Numpy arrays are always stored in contiguous blocks of memory with
> > uniform strides. The "CONTIGUOUS" flag actually means something
> > totally different, which is unfortunate, but in any case, "fancy
> > indexing" can't be done as a simple reindexing operation. It must make
> > a copy of the array. So what you're seeing is the flags of a fresh new
>
> > array, created from scratch (and numpy always creates arrays in C
> > order internally, though that is an implementation detail you should
> > not rely on).
>
> If you are correct that this is in fact a fresh new array, I really
> don't understand where the values of these flags. To recap:
>
> In [19]: x = N.zeros((3,2))
>
> In [20]: x.flags
> Out[20]:
>   C_CONTIGUOUS : True
>   F_CONTIGUOUS : False
>   OWNDATA : True
>   WRITEABLE : True
>   ALIGNED : True
>   UPDATEIFCOPY : False
>
> In [21]: x[:,[1,0]].flags
> Out[21]:
>   C_CONTIGUOUS : False
>   F_CONTIGUOUS : True
>   OWNDATA : False
>   WRITEABLE : True
>   ALIGNED : True
>   UPDATEIFCOPY : False
>
> So since x and x[:,[1,0]] are both new arrays, shouldn't their flags be
> identical? I'd expect at least C_CONTIGUOUS and OWNDATA to be True.


The contiguous refers to how stuff is layed out in memory. In this case it
appears that fancy indexing creates the new array by first copying column 1,
then column 2, so that the new array is indeed F_CONTIGUOUS. Assuming I
correctly understand the behaviour of the tostring argument, which is
debatable, that is indeed what happens.



Make that

The contiguous flags refer to how stuff is layed out in memory. In this case
it appears that fancy indexing creates the new array by first copying column
1, then column 0, so that the new array is indeed F_CONTIGUOUS. Assuming I
correctly understand the behaviour of the tostring argument, which is
debatable, that is indeed what happens.

Real programmers don't do writing.

Chuck
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Question about flags of fancy indexed array

[Charles R Harris]

On 5/23/07, Albert Strasheim <[EMAIL PROTECTED]> wrote:


Hello all

On Wed, 23 May 2007, Anne Archibald wrote:

> On 23/05/07, Albert Strasheim <[EMAIL PROTECTED]> wrote:
>
> > Consider the following example:
>
> First a comment: almost nobody needs to care how the data is stored
> internally. Try to avoid looking at the flags unless you're
> interfacing with a C library. The nice feature of numpy is that it
> hides all that junk - strides, contiguous storage, iteration, what
> have you - so that you don't have to deal with it.

As luck would have it, I am interfacing with a C library.

> > Is it correct that the F_CONTIGUOUS flag is set in the case of the
fancy
> > indexed x? I'm running NumPy 1.0.3.dev3792 here.
>
> Numpy arrays are always stored in contiguous blocks of memory with
> uniform strides. The "CONTIGUOUS" flag actually means something
> totally different, which is unfortunate, but in any case, "fancy
> indexing" can't be done as a simple reindexing operation. It must make
> a copy of the array. So what you're seeing is the flags of a fresh new
> array, created from scratch (and numpy always creates arrays in C
> order internally, though that is an implementation detail you should
> not rely on).

If you are correct that this is in fact a fresh new array, I really
don't understand where the values of these flags. To recap:

In [19]: x = N.zeros((3,2))

In [20]: x.flags
Out[20]:
  C_CONTIGUOUS : True
  F_CONTIGUOUS : False
  OWNDATA : True
  WRITEABLE : True
  ALIGNED : True
  UPDATEIFCOPY : False

In [21]: x[:,[1,0]].flags
Out[21]:
  C_CONTIGUOUS : False
  F_CONTIGUOUS : True
  OWNDATA : False
  WRITEABLE : True
  ALIGNED : True
  UPDATEIFCOPY : False

So since x and x[:,[1,0]] are both new arrays, shouldn't their flags be
identical? I'd expect at least C_CONTIGUOUS and OWNDATA to be True.



The contiguous refers to how stuff is layed out in memory. In this case it
appears that fancy indexing creates the new array by first copying column 1,
then column 2, so that the new array is indeed F_CONTIGUOUS. Assuming I
correctly understand the behaviour of the tostring argument, which is
debatable, that is indeed what happens.

In [28]: x = arange(6, dtype=int8).reshape(3,2)

In [29]: x
Out[29]:
array([[0, 1],
  [2, 3],
  [4, 5]], dtype=int8)

In [30]: y = x[:,[1,0]]

In [31]: y
Out[31]:
array([[1, 0],
  [3, 2],
  [5, 4]], dtype=int8)

In [32]: x.tostring('A')
Out[32]: '\x00\x01\x02\x03\x04\x05'

In [33]: y.tostring('A')
Out[33]: '\x01\x03\x05\x00\x02\x04'

Chuck
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Question about flags of fancy indexed array

[Anne Archibald]

On 23/05/07, Albert Strasheim <[EMAIL PROTECTED]> wrote:

> If you are correct that this is in fact a fresh new array, I really
> don't understand where the values of these flags. To recap:
>
> In [19]: x = N.zeros((3,2))
>
> In [20]: x.flags
> Out[20]:
>   C_CONTIGUOUS : True
>   F_CONTIGUOUS : False
>   OWNDATA : True
>   WRITEABLE : True
>   ALIGNED : True
>   UPDATEIFCOPY : False
>
> In [21]: x[:,[1,0]].flags
> Out[21]:
>   C_CONTIGUOUS : False
>   F_CONTIGUOUS : True
>   OWNDATA : False
>   WRITEABLE : True
>   ALIGNED : True
>   UPDATEIFCOPY : False
>
> So since x and x[:,[1,0]] are both new arrays, shouldn't their flags be
> identical? I'd expect at least C_CONTIGUOUS and OWNDATA to be True.

It looks like x[:,[1,0]] is done by fancy indexing on the first index
and then transposing. I haven't looked at the implementation, though.
If you need the result to be C-contiguous without further copying, you
can do:
x.transpose()[[1,0],:].transpose().flags
which is horrible. I wouldn't rely on it, though, I'd use
asconguousarray afterward.

Perhaps Travis could comment on whether it is true that numpy does
transpose like this when fancy indexing (at least, this flavour of
fancy indexing).

Anne
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Question about flags of fancy indexed array

[Stefan van der Walt]

On Wed, May 23, 2007 at 09:49:08AM -0400, Anne Archibald wrote:
> On 23/05/07, Albert Strasheim <[EMAIL PROTECTED]> wrote:
>
> > Is it correct that the F_CONTIGUOUS flag is set in the case of the fancy
> > indexed x? I'm running NumPy 1.0.3.dev3792 here.
> 
> Numpy arrays are always stored in contiguous blocks of memory with
> uniform strides. The "CONTIGUOUS" flag actually means something
> totally different, which is unfortunate, but in any case, "fancy
> indexing" can't be done as a simple reindexing operation. It must make
> a copy of the array. So what you're seeing is the flags of a fresh new
> array, created from scratch (and numpy always creates arrays in C
> order internally, though that is an implementation detail you should
> not rely on).

That still doesn't explain

In [41]: N.zeros((3,2))[:,[0,1]].flags
Out[41]: 
  C_CONTIGUOUS : False
  F_CONTIGUOUS : True <<<
  OWNDATA : False <<<
  WRITEABLE : True
  ALIGNED : True
  UPDATEIFCOPY : False

vs.

In [40]: N.zeros((3,2),order='F')[:,[0,1]].flags
Out[40]: 
  C_CONTIGUOUS : True
  F_CONTIGUOUS : False <<
  OWNDATA : False  <<
  WRITEABLE : True
  ALIGNED : True
  UPDATEIFCOPY : False

Maybe the Fortran-ordering quiz at

http://mentat.za.net/numpy/quiz

needs an update! :)

Cheers
Stéfan
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Question about flags of fancy indexed array

[Albert Strasheim]

Hello all

On Wed, 23 May 2007, Anne Archibald wrote:

> On 23/05/07, Albert Strasheim <[EMAIL PROTECTED]> wrote:
> 
> > Consider the following example:
> 
> First a comment: almost nobody needs to care how the data is stored
> internally. Try to avoid looking at the flags unless you're
> interfacing with a C library. The nice feature of numpy is that it
> hides all that junk - strides, contiguous storage, iteration, what
> have you - so that you don't have to deal with it.

As luck would have it, I am interfacing with a C library.
 
> > Is it correct that the F_CONTIGUOUS flag is set in the case of the fancy
> > indexed x? I'm running NumPy 1.0.3.dev3792 here.
> 
> Numpy arrays are always stored in contiguous blocks of memory with
> uniform strides. The "CONTIGUOUS" flag actually means something
> totally different, which is unfortunate, but in any case, "fancy
> indexing" can't be done as a simple reindexing operation. It must make
> a copy of the array. So what you're seeing is the flags of a fresh new
> array, created from scratch (and numpy always creates arrays in C
> order internally, though that is an implementation detail you should
> not rely on).

If you are correct that this is in fact a fresh new array, I really 
don't understand where the values of these flags. To recap:

In [19]: x = N.zeros((3,2))

In [20]: x.flags
Out[20]:
  C_CONTIGUOUS : True
  F_CONTIGUOUS : False
  OWNDATA : True
  WRITEABLE : True
  ALIGNED : True
  UPDATEIFCOPY : False

In [21]: x[:,[1,0]].flags
Out[21]:
  C_CONTIGUOUS : False
  F_CONTIGUOUS : True
  OWNDATA : False
  WRITEABLE : True
  ALIGNED : True
  UPDATEIFCOPY : False

So since x and x[:,[1,0]] are both new arrays, shouldn't their flags be 
identical? I'd expect at least C_CONTIGUOUS and OWNDATA to be True.

Thanks.

Cheers,

Albert
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Question about flags of fancy indexed array

[Anne Archibald]

On 23/05/07, Albert Strasheim <[EMAIL PROTECTED]> wrote:

> Consider the following example:

First a comment: almost nobody needs to care how the data is stored
internally. Try to avoid looking at the flags unless you're
interfacing with a C library. The nice feature of numpy is that it
hides all that junk - strides, contiguous storage, iteration, what
have you - so that you don't have to deal with it.

> Is it correct that the F_CONTIGUOUS flag is set in the case of the fancy
> indexed x? I'm running NumPy 1.0.3.dev3792 here.

Numpy arrays are always stored in contiguous blocks of memory with
uniform strides. The "CONTIGUOUS" flag actually means something
totally different, which is unfortunate, but in any case, "fancy
indexing" can't be done as a simple reindexing operation. It must make
a copy of the array. So what you're seeing is the flags of a fresh new
array, created from scratch (and numpy always creates arrays in C
order internally, though that is an implementation detail you should
not rely on).

Anne
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Question about flags of fancy indexed array

[Albert Strasheim]

Hello all

Consider the following example:

In [43]: x = N.zeros((3,2))

In [44]: x.flags
Out[44]:
  C_CONTIGUOUS : True
  F_CONTIGUOUS : False
  OWNDATA : True
  WRITEABLE : True
  ALIGNED : True
  UPDATEIFCOPY : False

In [45]: x[:,[1,0]].flags
Out[45]:
  C_CONTIGUOUS : False
  F_CONTIGUOUS : True
  OWNDATA : False
  WRITEABLE : True
  ALIGNED : True
  UPDATEIFCOPY : False

Is it correct that the F_CONTIGUOUS flag is set in the case of the fancy 
indexed x? I'm running NumPy 1.0.3.dev3792 here.

Cheers,

Albert 

___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion