[
https://issues.apache.org/jira/browse/DRILL-6230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16421542#comment-16421542
]
ASF GitHub Bot commented on DRILL-6230:
---------------------------------------
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/1161#discussion_r177846274
--- Diff:
exec/java-exec/src/test/java/org/apache/drill/test/rowSet/HyperRowSetImpl.java
---
@@ -45,8 +50,67 @@ public RowSetReader buildReader(HyperRowSet rowSet,
SelectionVector4 sv4) {
TupleMetadata schema = rowSet.schema();
HyperRowIndex rowIndex = new HyperRowIndex(sv4);
return new RowSetReaderImpl(schema, rowIndex,
- buildContainerChildren(rowSet.container(),
- new MetadataRetrieval(schema)));
+ buildContainerChildren(rowSet.container(), schema));
+ }
+ }
+
+ public static class HyperRowSetBuilderImpl implements HyperRowSetBuilder
{
+
+ private final BufferAllocator allocator;
+ private final List<VectorContainer> batches = new ArrayList<>();
+ private int totalRowCount;
+
+ public HyperRowSetBuilderImpl(BufferAllocator allocator) {
+ this.allocator = allocator;
+ }
+
+ @Override
+ public void addBatch(SingleRowSet rowSet) {
+ if (rowSet.rowCount() == 0) {
+ return;
+ }
+ if (rowSet.indirectionType() != SelectionVectorMode.NONE) {
+ throw new IllegalArgumentException("Batches must not have a
selection vector.");
+ }
+ batches.add(rowSet.container());
+ totalRowCount += rowSet.rowCount();
+ }
+
+ @Override
+ public void addBatch(VectorContainer container) {
+ if (container.getRecordCount() == 0) {
+ return;
+ }
+ if (container.getSchema().getSelectionVectorMode() !=
SelectionVectorMode.NONE) {
+ throw new IllegalArgumentException("Batches must not have a
selection vector.");
+ }
+ batches.add(container);
+ totalRowCount += container.getRecordCount();
+ }
+
+ @SuppressWarnings("resource")
+ @Override
+ public HyperRowSet build() throws SchemaChangeException {
+ SelectionVector4 sv4 = new SelectionVector4(allocator,
totalRowCount);
+ ExpandableHyperContainer hyperContainer = new
ExpandableHyperContainer();
+ for (VectorContainer container : batches) {
+ hyperContainer.addBatch(container);
+ }
+
+ // TODO: This has a bug. If the hyperset has two batches with unions,
+ // and the first union contains only VARCHAR, while the second
contains
--- End diff --
As noted in the remainder of the comment: "This is only a theoretical bug
as Drill does not support unions completely." That is, trying to make this case
work now would require us to write special code just to set up the case, as no
existing scan operators could ever produce this case. What we need is a more
general JIRA: either fix unions, or deprecate them. Having a half-baked feature
is quite hard to reason about.
> Extend row set readers to handle hyper vectors
> ----------------------------------------------
>
> Key: DRILL-6230
> URL: https://issues.apache.org/jira/browse/DRILL-6230
> Project: Apache Drill
> Issue Type: Improvement
> Reporter: Paul Rogers
> Assignee: Paul Rogers
> Priority: Major
> Fix For: 1.14.0
>
>
> The current row set readers have incomplete support for hyper-vectors. To add
> full support, we need an interface that supports either single batches or
> hyper batches. Accessing vectors in hyper batches differs depending on
> whether the vector is at the top level or is nested. See [this
> post|https://github.com/paul-rogers/drill/wiki/BH-Column-Readers] for
> details. Also includes a simpler reader template: replaces the original three
> classes with one, in parallel with the writers.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
