[jira] [Updated] (ARROW-2913) [Python] Exported buffers don't expose type information

2018-11-19 Thread Wes McKinney (JIRA) 


 [ 
https://issues.apache.org/jira/browse/ARROW-2913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wes McKinney updated ARROW-2913:

Fix Version/s: (was: 0.12.0)
   0.13.0

> [Python] Exported buffers don't expose type information
> ---
>
> Key: ARROW-2913
> URL: https://issues.apache.org/jira/browse/ARROW-2913
> Project: Apache Arrow
>  Issue Type: Improvement
>  Components: C++, Python
>Affects Versions: 0.10.0
>Reporter: Antoine Pitrou
>Priority: Major
> Fix For: 0.13.0
>
>
> Using the {{buffers()}} method on array gives you a list of buffers backing 
> the array, but those buffers lose typing information:
> {code:python}
> >>> a = pa.array(range(10))
> >>> a.type
> DataType(int64)
> >>> buffers = a.buffers()
> >>> [(memoryview(buf).format, memoryview(buf).shape) for buf in buffers]
> [('b', (2,)), ('b', (80,))]
> {code}
> Conversely, Numpy exposes type information in the Python buffer protocol:
> {code:python}
> >>> a = pa.array(range(10))
> >>> memoryview(a.to_numpy()).format
> 'l'
> >>> memoryview(a.to_numpy()).shape
> (10,)
> {code}
> Exposing type information on buffers could be important for third-party 
> systems, such as Dask/distributed, for type-based data compression when 
> serializing.
> Since our C++ buffers are not typed, it's not obvious how to solve this. 
> Should we return tensors instead?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (ARROW-2913) [Python] Exported buffers don't expose type information

2018-09-10 Thread Wes McKinney (JIRA) 


 [ 
https://issues.apache.org/jira/browse/ARROW-2913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wes McKinney updated ARROW-2913:

Fix Version/s: (was: 0.11.0)
   0.12.0

> [Python] Exported buffers don't expose type information
> ---
>
> Key: ARROW-2913
> URL: https://issues.apache.org/jira/browse/ARROW-2913
> Project: Apache Arrow
>  Issue Type: Improvement
>  Components: C++, Python
>Affects Versions: 0.10.0
>Reporter: Antoine Pitrou
>Priority: Major
> Fix For: 0.12.0
>
>
> Using the {{buffers()}} method on array gives you a list of buffers backing 
> the array, but those buffers lose typing information:
> {code:python}
> >>> a = pa.array(range(10))
> >>> a.type
> DataType(int64)
> >>> buffers = a.buffers()
> >>> [(memoryview(buf).format, memoryview(buf).shape) for buf in buffers]
> [('b', (2,)), ('b', (80,))]
> {code}
> Conversely, Numpy exposes type information in the Python buffer protocol:
> {code:python}
> >>> a = pa.array(range(10))
> >>> memoryview(a.to_numpy()).format
> 'l'
> >>> memoryview(a.to_numpy()).shape
> (10,)
> {code}
> Exposing type information on buffers could be important for third-party 
> systems, such as Dask/distributed, for type-based data compression when 
> serializing.
> Since our C++ buffers are not typed, it's not obvious how to solve this. 
> Should we return tensors instead?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (ARROW-2913) [Python] Exported buffers don't expose type information

2018-08-09 Thread Wes McKinney (JIRA) 


 [ 
https://issues.apache.org/jira/browse/ARROW-2913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wes McKinney updated ARROW-2913:

Fix Version/s: 0.11.0

> [Python] Exported buffers don't expose type information
> ---
>
> Key: ARROW-2913
> URL: https://issues.apache.org/jira/browse/ARROW-2913
> Project: Apache Arrow
>  Issue Type: Improvement
>  Components: C++, Python
>Affects Versions: 0.10.0
>Reporter: Antoine Pitrou
>Priority: Major
> Fix For: 0.11.0
>
>
> Using the {{buffers()}} method on array gives you a list of buffers backing 
> the array, but those buffers lose typing information:
> {code:python}
> >>> a = pa.array(range(10))
> >>> a.type
> DataType(int64)
> >>> buffers = a.buffers()
> >>> [(memoryview(buf).format, memoryview(buf).shape) for buf in buffers]
> [('b', (2,)), ('b', (80,))]
> {code}
> Conversely, Numpy exposes type information in the Python buffer protocol:
> {code:python}
> >>> a = pa.array(range(10))
> >>> memoryview(a.to_numpy()).format
> 'l'
> >>> memoryview(a.to_numpy()).shape
> (10,)
> {code}
> Exposing type information on buffers could be important for third-party 
> systems, such as Dask/distributed, for type-based data compression when 
> serializing.
> Since our C++ buffers are not typed, it's not obvious how to solve this. 
> Should we return tensors instead?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)