[jira] [Commented] (ARROW-7048) [Java] Support for combining multiple vectors under VectorSchemaRoot
[
https://issues.apache.org/jira/browse/ARROW-7048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17046623#comment-17046623
]
Liya Fan commented on ARROW-7048:
-
[~yogeshtewari] Sorry for the long wait. We have provided a PR for this issue.
Would you please take a look, and check if it is what you want?
> [Java] Support for combining multiple vectors under VectorSchemaRoot
>
>
> Key: ARROW-7048
> URL: https://issues.apache.org/jira/browse/ARROW-7048
> Project: Apache Arrow
> Issue Type: New Feature
> Components: Java
>Reporter: Yogesh Tewari
>Assignee: Liya Fan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hi,
>
> pyarrow.Table.combine_chunks provides a nice functionality of combining
> multiple batch records under a single pyarrow.Table.
>
> I am currently working on a downstream application which reads data from
> BigQuery. BigQuery storage api supports data output in Arrow format but
> streams data in many batches of size 1024 or less number of rows.
> It would be really nice to have Arrow Java api provide this functionality
> under an abstraction like VectorSchemaRoot.
> After getting guidance from [~emkornfi...@gmail.com], I tried to write my own
> implementation by copying data vector by vector using TransferPair's
> copyValueSafe
> But, unless I am missing some thing obvious, turns out it only copies one
> value at a time. That means a lot of looping trying copyValueSafe millions of
> rows from source vector index to target vector index. Ideally I would want to
> concatenate/link the underlying buffers rather than copying one cell at a
> time.
>
> Eg, if I have :
> {code:java}
> List batchList = new ArrayList<>();
> try (ArrowStreamReader reader = new ArrowStreamReader(new
> ByteArrayInputStream(out.toByteArray()), allocator)) {
> Schema schema = reader.getVectorSchemaRoot().getSchema();
> for (int i = 0; i < 5; i++) {
> // This will be loaded with new values on every call to loadNextBatch
> VectorSchemaRoot readBatch = reader.getVectorSchemaRoot();
> reader.loadNextBatch();
> batchList.add(readBatch);
> }
> }
> //VectorSchemaRoot.combineChunks(batchList, newVectorSchemaRoot);{code}
>
> A method like VectorSchemaRoot.combineChunks(List)?
> I did read the VectorSchemaRoot discussion on
> https://issues.apache.org/jira/browse/ARROW-6896 and am not sure if its the
> right thing to use here.
>
>
> PS. Feel free to update the title of this feature request with more
> appropriate wordings.
>
> Cheers,
> Yogesh
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ARROW-7048) [Java] Support for combining multiple vectors under VectorSchemaRoot
[
https://issues.apache.org/jira/browse/ARROW-7048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16969016#comment-16969016
]
Liya Fan commented on ARROW-7048:
-
[~emkornfi...@gmail.com] Agreed. Adding a constant to each offset is more
efficient.
> [Java] Support for combining multiple vectors under VectorSchemaRoot
>
>
> Key: ARROW-7048
> URL: https://issues.apache.org/jira/browse/ARROW-7048
> Project: Apache Arrow
> Issue Type: New Feature
> Components: Java
>Reporter: Yogesh Tewari
>Assignee: Liya Fan
>Priority: Major
>
> Hi,
>
> pyarrow.Table.combine_chunks provides a nice functionality of combining
> multiple batch records under a single pyarrow.Table.
>
> I am currently working on a downstream application which reads data from
> BigQuery. BigQuery storage api supports data output in Arrow format but
> streams data in many batches of size 1024 or less number of rows.
> It would be really nice to have Arrow Java api provide this functionality
> under an abstraction like VectorSchemaRoot.
> After getting guidance from [~emkornfi...@gmail.com], I tried to write my own
> implementation by copying data vector by vector using TransferPair's
> copyValueSafe
> But, unless I am missing some thing obvious, turns out it only copies one
> value at a time. That means a lot of looping trying copyValueSafe millions of
> rows from source vector index to target vector index. Ideally I would want to
> concatenate/link the underlying buffers rather than copying one cell at a
> time.
>
> Eg, if I have :
> {code:java}
> List batchList = new ArrayList<>();
> try (ArrowStreamReader reader = new ArrowStreamReader(new
> ByteArrayInputStream(out.toByteArray()), allocator)) {
> Schema schema = reader.getVectorSchemaRoot().getSchema();
> for (int i = 0; i < 5; i++) {
> // This will be loaded with new values on every call to loadNextBatch
> VectorSchemaRoot readBatch = reader.getVectorSchemaRoot();
> reader.loadNextBatch();
> batchList.add(readBatch);
> }
> }
> //VectorSchemaRoot.combineChunks(batchList, newVectorSchemaRoot);{code}
>
> A method like VectorSchemaRoot.combineChunks(List)?
> I did read the VectorSchemaRoot discussion on
> https://issues.apache.org/jira/browse/ARROW-6896 and am not sure if its the
> right thing to use here.
>
>
> PS. Feel free to update the title of this feature request with more
> appropriate wordings.
>
> Cheers,
> Yogesh
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ARROW-7048) [Java] Support for combining multiple vectors under VectorSchemaRoot
[
https://issues.apache.org/jira/browse/ARROW-7048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16968998#comment-16968998
]
Micah Kornfield commented on ARROW-7048:
"For VariableWidthVectors, we need to transform the offset buffer to a delta
buffer, do the copy, and then transform the delta buffer back to a partial sum
buffer. This may involve another feature discussed in ARROW-6394."
I don't think a transformation between the two is necessary. Would you simply
need to add a constant to each set of offset (i.e. translation back and forth
is more costly then necessary).
> [Java] Support for combining multiple vectors under VectorSchemaRoot
>
>
> Key: ARROW-7048
> URL: https://issues.apache.org/jira/browse/ARROW-7048
> Project: Apache Arrow
> Issue Type: New Feature
> Components: Java
>Reporter: Yogesh Tewari
>Assignee: Liya Fan
>Priority: Major
>
> Hi,
>
> pyarrow.Table.combine_chunks provides a nice functionality of combining
> multiple batch records under a single pyarrow.Table.
>
> I am currently working on a downstream application which reads data from
> BigQuery. BigQuery storage api supports data output in Arrow format but
> streams data in many batches of size 1024 or less number of rows.
> It would be really nice to have Arrow Java api provide this functionality
> under an abstraction like VectorSchemaRoot.
> After getting guidance from [~emkornfi...@gmail.com], I tried to write my own
> implementation by copying data vector by vector using TransferPair's
> copyValueSafe
> But, unless I am missing some thing obvious, turns out it only copies one
> value at a time. That means a lot of looping trying copyValueSafe millions of
> rows from source vector index to target vector index. Ideally I would want to
> concatenate/link the underlying buffers rather than copying one cell at a
> time.
>
> Eg, if I have :
> {code:java}
> List batchList = new ArrayList<>();
> try (ArrowStreamReader reader = new ArrowStreamReader(new
> ByteArrayInputStream(out.toByteArray()), allocator)) {
> Schema schema = reader.getVectorSchemaRoot().getSchema();
> for (int i = 0; i < 5; i++) {
> // This will be loaded with new values on every call to loadNextBatch
> VectorSchemaRoot readBatch = reader.getVectorSchemaRoot();
> reader.loadNextBatch();
> batchList.add(readBatch);
> }
> }
> //VectorSchemaRoot.combineChunks(batchList, newVectorSchemaRoot);{code}
>
> A method like VectorSchemaRoot.combineChunks(List)?
> I did read the VectorSchemaRoot discussion on
> https://issues.apache.org/jira/browse/ARROW-6896 and am not sure if its the
> right thing to use here.
>
>
> PS. Feel free to update the title of this feature request with more
> appropriate wordings.
>
> Cheers,
> Yogesh
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ARROW-7048) [Java] Support for combining multiple vectors under VectorSchemaRoot
[
https://issues.apache.org/jira/browse/ARROW-7048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16968099#comment-16968099
]
Yogesh Tewari commented on ARROW-7048:
--
Sure [~fan_li_ya] . Go for it.
Sorry I couldn't respond earlier, but thanks for the comments. It makes sense.
> [Java] Support for combining multiple vectors under VectorSchemaRoot
>
>
> Key: ARROW-7048
> URL: https://issues.apache.org/jira/browse/ARROW-7048
> Project: Apache Arrow
> Issue Type: New Feature
> Components: Java
>Reporter: Yogesh Tewari
>Priority: Major
>
> Hi,
>
> pyarrow.Table.combine_chunks provides a nice functionality of combining
> multiple batch records under a single pyarrow.Table.
>
> I am currently working on a downstream application which reads data from
> BigQuery. BigQuery storage api supports data output in Arrow format but
> streams data in many batches of size 1024 or less number of rows.
> It would be really nice to have Arrow Java api provide this functionality
> under an abstraction like VectorSchemaRoot.
> After getting guidance from [~emkornfi...@gmail.com], I tried to write my own
> implementation by copying data vector by vector using TransferPair's
> copyValueSafe
> But, unless I am missing some thing obvious, turns out it only copies one
> value at a time. That means a lot of looping trying copyValueSafe millions of
> rows from source vector index to target vector index. Ideally I would want to
> concatenate/link the underlying buffers rather than copying one cell at a
> time.
>
> Eg, if I have :
> {code:java}
> List batchList = new ArrayList<>();
> try (ArrowStreamReader reader = new ArrowStreamReader(new
> ByteArrayInputStream(out.toByteArray()), allocator)) {
> Schema schema = reader.getVectorSchemaRoot().getSchema();
> for (int i = 0; i < 5; i++) {
> // This will be loaded with new values on every call to loadNextBatch
> VectorSchemaRoot readBatch = reader.getVectorSchemaRoot();
> reader.loadNextBatch();
> batchList.add(readBatch);
> }
> }
> //VectorSchemaRoot.combineChunks(batchList, newVectorSchemaRoot);{code}
>
> A method like VectorSchemaRoot.combineChunks(List)?
> I did read the VectorSchemaRoot discussion on
> https://issues.apache.org/jira/browse/ARROW-6896 and am not sure if its the
> right thing to use here.
>
>
> PS. Feel free to update the title of this feature request with more
> appropriate wordings.
>
> Cheers,
> Yogesh
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ARROW-7048) [Java] Support for combining multiple vectors under VectorSchemaRoot
[
https://issues.apache.org/jira/browse/ARROW-7048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16968023#comment-16968023
]
Liya Fan commented on ARROW-7048:
-
[~yogeshtewari] I notice that the assignee of this issue is left empty.
If you don't mind, may I try to provide a solution to it?
> [Java] Support for combining multiple vectors under VectorSchemaRoot
>
>
> Key: ARROW-7048
> URL: https://issues.apache.org/jira/browse/ARROW-7048
> Project: Apache Arrow
> Issue Type: New Feature
> Components: Java
>Reporter: Yogesh Tewari
>Priority: Major
>
> Hi,
>
> pyarrow.Table.combine_chunks provides a nice functionality of combining
> multiple batch records under a single pyarrow.Table.
>
> I am currently working on a downstream application which reads data from
> BigQuery. BigQuery storage api supports data output in Arrow format but
> streams data in many batches of size 1024 or less number of rows.
> It would be really nice to have Arrow Java api provide this functionality
> under an abstraction like VectorSchemaRoot.
> After getting guidance from [~emkornfi...@gmail.com], I tried to write my own
> implementation by copying data vector by vector using TransferPair's
> copyValueSafe
> But, unless I am missing some thing obvious, turns out it only copies one
> value at a time. That means a lot of looping trying copyValueSafe millions of
> rows from source vector index to target vector index. Ideally I would want to
> concatenate/link the underlying buffers rather than copying one cell at a
> time.
>
> Eg, if I have :
> {code:java}
> List batchList = new ArrayList<>();
> try (ArrowStreamReader reader = new ArrowStreamReader(new
> ByteArrayInputStream(out.toByteArray()), allocator)) {
> Schema schema = reader.getVectorSchemaRoot().getSchema();
> for (int i = 0; i < 5; i++) {
> // This will be loaded with new values on every call to loadNextBatch
> VectorSchemaRoot readBatch = reader.getVectorSchemaRoot();
> reader.loadNextBatch();
> batchList.add(readBatch);
> }
> }
> //VectorSchemaRoot.combineChunks(batchList, newVectorSchemaRoot);{code}
>
> A method like VectorSchemaRoot.combineChunks(List)?
> I did read the VectorSchemaRoot discussion on
> https://issues.apache.org/jira/browse/ARROW-6896 and am not sure if its the
> right thing to use here.
>
>
> PS. Feel free to update the title of this feature request with more
> appropriate wordings.
>
> Cheers,
> Yogesh
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ARROW-7048) [Java] Support for combining multiple vectors under VectorSchemaRoot
[
https://issues.apache.org/jira/browse/ARROW-7048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16966345#comment-16966345
]
Liya Fan commented on ARROW-7048:
-
[~yogeshtewari] Thanks a lot for opening this issue.
I think your scenario represents some general requirements. IMO, to support
your requirement, some fundamental primitives needs to be supported.
I can think of two possible ways of solving the problem:
1. support memory address linking at the ArrowBuf level. This may be
impractical, as ArrowBuf is extremely performance critical.
2. support bulk copying/appending API, with high performance. In the ideal
case, a single copying should accomplish the task for each underlying ArrowBuf.
The difficulty for this is that the solution must be provided case by case. For
example, for FixedWidthVectors, we can extend the buffer, and perform the
copying directly. For VariableWidthVectors, we need to transform the offset
buffer to a delta buffer, do the copy, and then transform the delta buffer back
to a partial sum buffer. This may involve another feature discussed in
ARROW-6394.
Anyway, I think the feature is useful, but it would be difficult to support it
in a single step.
> [Java] Support for combining multiple vectors under VectorSchemaRoot
>
>
> Key: ARROW-7048
> URL: https://issues.apache.org/jira/browse/ARROW-7048
> Project: Apache Arrow
> Issue Type: New Feature
> Components: Java
>Reporter: Yogesh Tewari
>Priority: Major
>
> Hi,
>
> pyarrow.Table.combine_chunks provides a nice functionality of combining
> multiple batch records under a single pyarrow.Table.
>
> I am currently working on a downstream application which reads data from
> BigQuery. BigQuery storage api supports data output in Arrow format but
> streams data in many batches of size 1024 or less number of rows.
> It would be really nice to have Arrow Java api provide this functionality
> under an abstraction like VectorSchemaRoot.
> After getting guidance from [~emkornfi...@gmail.com], I tried to write my own
> implementation by copying data vector by vector using TransferPair's
> copyValueSafe
> But, unless I am missing some thing obvious, turns out it only copies one
> value at a time. That means a lot of looping trying copyValueSafe millions of
> rows from source vector index to target vector index. Ideally I would want to
> concatenate/link the underlying buffers rather than copying one cell at a
> time.
>
> Eg, if I have :
> {code:java}
> List batchList = new ArrayList<>();
> try (ArrowStreamReader reader = new ArrowStreamReader(new
> ByteArrayInputStream(out.toByteArray()), allocator)) {
> Schema schema = reader.getVectorSchemaRoot().getSchema();
> for (int i = 0; i < 5; i++) {
> // This will be loaded with new values on every call to loadNextBatch
> VectorSchemaRoot readBatch = reader.getVectorSchemaRoot();
> reader.loadNextBatch();
> batchList.add(readBatch);
> }
> }
> //VectorSchemaRoot.combineChunks(batchList, newVectorSchemaRoot);{code}
>
> A method like VectorSchemaRoot.combineChunks(List)?
> I did read the VectorSchemaRoot discussion on
> https://issues.apache.org/jira/browse/ARROW-6896 and am not sure if its the
> right thing to use here.
>
>
> PS. Feel free to update the title of this feature request with more
> appropriate wordings.
>
> Cheers,
> Yogesh
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
