GitHub user paul-rogers opened a pull request: https://github.com/apache/drill/pull/1244
DRILL-6373: Refactor Result Set Loader for Union, List support This PR builds on the previous refactoring of the column accessors to prepare for Union, (non-repeated) List and Repeated List support. The PR includes four closely related changes divided across four commits: ### Correct the Type of the Data Vector in a Nullable Vector The nullable vectors contain a "bits" vector and a "data" vector. The data vector has historically been created using the same `MaterializedField` as the nullable vector, meaning that the data vector is labeled as "nullable" even though it has no bits vector. This PR creates a clone `MaterializedField` with the same name as the outer nullable vector, but with a Required type. This change ensures that the overflow logic works correctly as it uses the vector metadata (in the `MaterializedField`) to know what kind of vector to create for the "lookahead" vector. ### Result Set Loader Refactor The second commit pretty much just rearranges the deck chairs in a way that we an slot in the new types in the next PR. The need for the changes can be seen in the full code set (the union and list support was pulled out for this PR.) A union is a container, like a map, so the tuple state was refactored to create a common parent container state. List and unions are very complex to build, so the code to build the internal workings of each vector was pulled out into a separate builder class. ### Projection Handling and the Vector Cache Previous versions of the result set loader handled projection and a cache for vectors reused across readers in the same Scan operator. Once we introduce nested maps, projection within maps, unions and lists, projection gets much more complex, as does vector caching. This PR adds logic to support projection and vector caching to any arbitrary level of maps. It turns out that handling projection of an entire map, and projection of fields within maps, is far more complex than you'd think, requiring quite a bit of internal state to keep everything straight. The result is that we can now handle a map `m` with three fields `{a, b, c}` and project just one of them, `m.a`, say. Further, Drill allows projection of non-existent columns. So, we might ask for field `m.d` which does not exist in the above map. The projection mechanism handles this case as well, creating the right kind of null column. ### Unit Tests New tests are added to exercise the projection and cache mechanisms. Existing tests were updated for the changes made in the refactoring. ### Reference Design All of this work is done in support of the overall "batch sizing" project explained [here](https://github.com/paul-rogers/drill/wiki/Batch-Handling-Upgrades). You can merge this pull request into a Git repository by running: $ git pull https://github.com/paul-rogers/drill DRILL-6373 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/drill/pull/1244.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1244 ---- commit 7df4280f3011862b84b43240c8a07e0bf019745d Author: Paul Rogers <progers@...> Date: 2018-05-01T02:10:53Z DRILL-6373: Fix nullable vector data vector type Fixes the type of the data vector within a nullable vector. The data vector is Required (has no bits vector.) Accurate metadata is required for proper overflow handling in the result set loader. commit 9496ef681f19f03ccd735f2c9b18f6d914eae3e2 Author: Paul Rogers <progers@...> Date: 2018-05-01T02:15:33Z DRILL-6373: Refactor result set loader commit 74675436ae1efdf66deafeaa27b281d169e274ad Author: Paul Rogers <progers@...> Date: 2018-05-01T02:16:48Z DRILL-6373: Revised projection & vector cache commit 04598c0dbdbbff5ecbc2f89d02b14f66982f86bd Author: Paul Rogers <progers@...> Date: 2018-05-01T02:17:22Z DRILL-6373: Revised & added unit tests ---- ---