[jira] [Updated] (DRILL-5514) Enhance VectorContainer to merge two row sets
[
https://issues.apache.org/jira/browse/DRILL-5514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Rogers updated DRILL-5514:
---
Description:
Consider the concept of a "record batch" in Drill. On the one hand, one can
envision a record batch as a stack of records:
{code}
| a1 | b1 | c1 |
| a2 | b2 | c2 |
{code}
But, Drill is columnar. So a record batch is really a "bundle" of vectors:
{code}
| a1 || b1 || c1 |
| a2 || b2 || c2 |
{code}
There are times when it is handy to build up a record batch as a merge of two
different vector bundles:
{code}
-- bundle 1 ---- bundle 2 --
| a1 || b1 || c1 |
| a2 || b2 || c2 |
{code}
For example, consider a reader. The reader implementation might read columns
(a, b) from a file, say. Then, the "{{ScanBatch}}" might add (c) as an implicit
vector (the file name, say.) The merged set of vectors comprises the final
schema: (a, b, c).
This ticket asks for the code to do the merge:
* Merge two schemas A = (a, b), B = (c) to create schema C = (a, b, c).
* Merge two vector containers C1 and C2 to create a new container, C3, that
holds the merger of the vectors from the first two.
Clearly, the merge only makes sense if:
* The two input containers have the same row count, and
* The columns in each input container are distinct.
Because this feature is also useful for tests, add the merge to the "row set"
tools also.
was:
Consider the concept of a "record batch" in Drill. On the one hand, one can
envision a record batch as a stack of records:
{code}
| a1 | b1 | c1 |
| a2 | b2 | c2 |
{code}
But, Drill is columnar. So a record batch is really a "bundle" of vectors:
{code}
| a1 || b1 || c1 |
| a2 || b2 || c2 |
{code}
There are times when it is handy to build up a record batch as a merge of two
different vector bundles:
{code}
-- bundle 1 ---- bundle 2 --
| a1 || b1 || c1 |
| a2 || b2 || c2 |
{code}
For example, consider a reader. The reader implementation might read columns
(a, b) from a file, say. Then, the "{{ScanBatch}}" might add (c) as an implicit
vector (the file name, say.) The merged set of vectors comprises the final
schema: (a, b, c).
This ticket asks for the code to do the merge:
* Merge two schemas A = (a, b), B = (c) to create schema C = (a, b, c).
* Merge two vector containers C1 and C2 to create a new container, C3, that
holds the merger of the vectors from the first two.
> Enhance VectorContainer to merge two row sets
> -
>
> Key: DRILL-5514
> URL: https://issues.apache.org/jira/browse/DRILL-5514
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.10.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
> Fix For: 1.11.0
>
>
> Consider the concept of a "record batch" in Drill. On the one hand, one can
> envision a record batch as a stack of records:
> {code}
> | a1 | b1 | c1 |
>
> | a2 | b2 | c2 |
> {code}
> But, Drill is columnar. So a record batch is really a "bundle" of vectors:
> {code}
> | a1 || b1 || c1 |
> | a2 || b2 || c2 |
> {code}
> There are times when it is handy to build up a record batch as a merge of two
> different vector bundles:
> {code}
> -- bundle 1 ---- bundle 2 --
> | a1 || b1 || c1 |
> | a2 || b2 || c2 |
> {code}
> For example, consider a reader. The reader implementation might read columns
> (a, b) from a file, say. Then, the "{{ScanBatch}}" might add (c) as an
> implicit vector (the file name, say.) The merged set of vectors comprises the
> final schema: (a, b, c).
> This ticket asks for the code to do the merge:
> * Merge two schemas A = (a, b), B = (c) to create schema C = (a, b, c).
> * Merge two vector containers C1 and C2 to create a new container, C3, that
> holds the merger of the vectors from the first two.
> Clearly, the merge only makes sense if:
> * The two input containers have the same row count, and
> * The columns in each input container are distinct.
> Because this feature is also useful for tests, add the merge to the "row set"
> tools also.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (DRILL-5514) Enhance VectorContainer to merge two row sets
[
https://issues.apache.org/jira/browse/DRILL-5514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Rogers updated DRILL-5514:
---
Reviewer: Sorabh Hamirwasia
> Enhance VectorContainer to merge two row sets
> -
>
> Key: DRILL-5514
> URL: https://issues.apache.org/jira/browse/DRILL-5514
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.10.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
> Fix For: 1.11.0
>
>
> Consider the concept of a "record batch" in Drill. On the one hand, one can
> envision a record batch as a stack of records:
> {code}
> | a1 | b1 | c1 |
>
> | a2 | b2 | c2 |
> {code}
> But, Drill is columnar. So a record batch is really a "bundle" of vectors:
> {code}
> | a1 || b1 || c1 |
> | a2 || b2 || c2 |
> {code}
> There are times when it is handy to build up a record batch as a merge of two
> different vector bundles:
> {code}
> -- bundle 1 ---- bundle 2 --
> | a1 || b1 || c1 |
> | a2 || b2 || c2 |
> {code}
> For example, consider a reader. The reader implementation might read columns
> (a, b) from a file, say. Then, the "{{ScanBatch}}" might add (c) as an
> implicit vector (the file name, say.) The merged set of vectors comprises the
> final schema: (a, b, c).
> This ticket asks for the code to do the merge:
> * Merge two schemas A = (a, b), B = (c) to create schema C = (a, b, c).
> * Merge two vector containers C1 and C2 to create a new container, C3, that
> holds the merger of the vectors from the first two.
> Clearly, the merge only makes sense if:
> * The two input containers have the same row count, and
> * The columns in each input container are distinct.
> Because this feature is also useful for tests, add the merge to the "row set"
> tools also.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (DRILL-5514) Enhance VectorContainer to merge two row sets
[
https://issues.apache.org/jira/browse/DRILL-5514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthikeyan Manivannan updated DRILL-5514:
--
Reviewer: Karthikeyan Manivannan (was: Sorabh Hamirwasia)
> Enhance VectorContainer to merge two row sets
> -
>
> Key: DRILL-5514
> URL: https://issues.apache.org/jira/browse/DRILL-5514
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.10.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
> Fix For: 1.11.0
>
>
> Consider the concept of a "record batch" in Drill. On the one hand, one can
> envision a record batch as a stack of records:
> {code}
> | a1 | b1 | c1 |
>
> | a2 | b2 | c2 |
> {code}
> But, Drill is columnar. So a record batch is really a "bundle" of vectors:
> {code}
> | a1 || b1 || c1 |
> | a2 || b2 || c2 |
> {code}
> There are times when it is handy to build up a record batch as a merge of two
> different vector bundles:
> {code}
> -- bundle 1 ---- bundle 2 --
> | a1 || b1 || c1 |
> | a2 || b2 || c2 |
> {code}
> For example, consider a reader. The reader implementation might read columns
> (a, b) from a file, say. Then, the "{{ScanBatch}}" might add (c) as an
> implicit vector (the file name, say.) The merged set of vectors comprises the
> final schema: (a, b, c).
> This ticket asks for the code to do the merge:
> * Merge two schemas A = (a, b), B = (c) to create schema C = (a, b, c).
> * Merge two vector containers C1 and C2 to create a new container, C3, that
> holds the merger of the vectors from the first two.
> Clearly, the merge only makes sense if:
> * The two input containers have the same row count, and
> * The columns in each input container are distinct.
> Because this feature is also useful for tests, add the merge to the "row set"
> tools also.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (DRILL-5514) Enhance VectorContainer to merge two row sets
[
https://issues.apache.org/jira/browse/DRILL-5514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Rogers updated DRILL-5514:
---
Labels: ready-to-commit (was: )
> Enhance VectorContainer to merge two row sets
> -
>
> Key: DRILL-5514
> URL: https://issues.apache.org/jira/browse/DRILL-5514
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.10.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
> Labels: ready-to-commit
> Fix For: 1.11.0
>
>
> Consider the concept of a "record batch" in Drill. On the one hand, one can
> envision a record batch as a stack of records:
> {code}
> | a1 | b1 | c1 |
>
> | a2 | b2 | c2 |
> {code}
> But, Drill is columnar. So a record batch is really a "bundle" of vectors:
> {code}
> | a1 || b1 || c1 |
> | a2 || b2 || c2 |
> {code}
> There are times when it is handy to build up a record batch as a merge of two
> different vector bundles:
> {code}
> -- bundle 1 ---- bundle 2 --
> | a1 || b1 || c1 |
> | a2 || b2 || c2 |
> {code}
> For example, consider a reader. The reader implementation might read columns
> (a, b) from a file, say. Then, the "{{ScanBatch}}" might add (c) as an
> implicit vector (the file name, say.) The merged set of vectors comprises the
> final schema: (a, b, c).
> This ticket asks for the code to do the merge:
> * Merge two schemas A = (a, b), B = (c) to create schema C = (a, b, c).
> * Merge two vector containers C1 and C2 to create a new container, C3, that
> holds the merger of the vectors from the first two.
> Clearly, the merge only makes sense if:
> * The two input containers have the same row count, and
> * The columns in each input container are distinct.
> Because this feature is also useful for tests, add the merge to the "row set"
> tools also.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
