parthchandra commented on a change in pull request #1401: DRILL-6616: Batch
Processing for Lateral/Unnest
URL: https://github.com/apache/drill/pull/1401#discussion_r205897124
##########
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/unnest/UnnestImpl.java
##########
@@ -48,24 +51,47 @@
private static final Logger logger =
LoggerFactory.getLogger(UnnestImpl.class);
private ImmutableList<TransferPair> transfers;
- private LateralContract lateral; // corresponding lateral Join (or other
operator implementing the Lateral Contract)
private SelectionVectorMode svMode;
private RepeatedValueVector fieldToUnnest;
private RepeatedValueVector.RepeatedAccessor accessor;
private RecordBatch outgoing;
+ private IntVector rowIdVector ; // Allocated and owned by the
UnnestRecordBatch
+ private IntVector.Mutator rowIdVectorMutator;
+
/**
* The output batch limit starts at OUTPUT_ROW_COUNT, but may be decreased
* if records are found to be large.
*/
private int outputLimit = ValueVector.MAX_ROW_COUNT;
- // The index in the unnest column that is being processed.We start at zero
and continue until
+
+ /**
+ * We maintain three indexes
+ *
+ *
+ *
+ valueIndex 0 1 2 3
+ |- - - - -|- - - -|- -|- - - -|
+ | | | | | | | | | | | | | | | |
+ |- - - - -|- - - -|- -|- - - -|
+ innerValueIndex 0 1 2 3 4 0 1 2 3 0 1 0 1 2 3 |
+ runningInnerValueIndex 0 1 2 3 4 5 6 7 8 9 ...
+ *
+ *
+ *
+ */
+ private int valueIndex; // index in the incoming record being processed
+ // The index in the unnest column that is being processed. We start at zero
and continue until
// InnerValueCount is reached or if the batch limit is reached
// this allows for groups to be written between batches if we run out of
space, for cases where we have finished
// a batch on the boundary it will be set to 0
Review comment:
Sure. That's why I did the picture. It's so much easier than saying it in
words. :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
