[jira] [Commented] (ARROW-6971) [Rust] Replace "RecordBatchReader" with "BatchIterator"
[
https://issues.apache.org/jira/browse/ARROW-6971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16958012#comment-16958012
]
Paddy Horan commented on ARROW-6971:
Ahh, ok then.
> [Rust] Replace "RecordBatchReader" with "BatchIterator"
> ---
>
> Key: ARROW-6971
> URL: https://issues.apache.org/jira/browse/ARROW-6971
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Rust, Rust - DataFusion
>Affects Versions: 0.15.0
>Reporter: Paddy Horan
>Assignee: Paddy Horan
>Priority: Minor
> Fix For: 1.0.0
>
>
> As part of the recent reader work we introduced
> {code:java}
> // arrow::record_batch::RecordBatchReader{code}
> but in datafusion we have
> {code:java}
> // datafusion::physical_plan::BatchIterator
> {code}
> These two trait are almost identical (BatchIterator implements Send + Sync
> whereas RecordBatchReader does not). I propose we replace RecordBatchReader
> with BatchIterator (i.e. move it to arrow as it's generally useful outside of
> datafusion) and update parquet and data fusion accordingly.
> [~andygrove] [~liurenjie1024] do you see any issues with this?
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ARROW-6971) [Rust] Replace "RecordBatchReader" with "BatchIterator"
[
https://issues.apache.org/jira/browse/ARROW-6971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16957506#comment-16957506
]
Renjie Liu commented on ARROW-6971:
---
I have discussed with [~andygrove] . RecordBatchReader can't impl Sync+Send
because it used some unsafe techniques.
> [Rust] Replace "RecordBatchReader" with "BatchIterator"
> ---
>
> Key: ARROW-6971
> URL: https://issues.apache.org/jira/browse/ARROW-6971
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Rust, Rust - DataFusion
>Affects Versions: 0.15.0
>Reporter: Paddy Horan
>Assignee: Paddy Horan
>Priority: Minor
> Fix For: 1.0.0
>
>
> As part of the recent reader work we introduced
> {code:java}
> // arrow::record_batch::RecordBatchReader{code}
> but in datafusion we have
> {code:java}
> // datafusion::physical_plan::BatchIterator
> {code}
> These two trait are almost identical (BatchIterator implements Send + Sync
> whereas RecordBatchReader does not). I propose we replace RecordBatchReader
> with BatchIterator (i.e. move it to arrow as it's generally useful outside of
> datafusion) and update parquet and data fusion accordingly.
> [~andygrove] [~liurenjie1024] do you see any issues with this?
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
