[
https://issues.apache.org/jira/browse/DRILL-6223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16421106#comment-16421106
]
salim achouche commented on DRILL-6223:
---------------------------------------
Thanks Paul for your feedback!
I wanted to bring to your attention another implementation detail that we also
need to pay a closer look at; the current implementation of a column with all
nulls is not cheap:
* Consider a batch of 32k rows
* A VV with null integer values will require 32kb (bits) + 32kb * 4 = 160kb
* Each missing column will require that much memory per mini-fragment
* This is unless (similarly to the implicit columns) we optimize the VV
storage representation or / and push the column preservation to higher layers
such as the client or foreman
* I understand that handling few missing columns this way is fine but this
will not be the case if we are talking about dozens of such columns especially
that we are now ramping up our support to operator based batch sizing
> Drill fails on Schema changes
> ------------------------------
>
> Key: DRILL-6223
> URL: https://issues.apache.org/jira/browse/DRILL-6223
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Relational Operators
> Affects Versions: 1.10.0, 1.12.0
> Reporter: salim achouche
> Assignee: salim achouche
> Priority: Major
> Fix For: 1.14.0
>
>
> Drill Query Failing when selecting all columns from a Complex Nested Data
> File (Parquet) Set). There are differences in Schema among the files:
> * The Parquet files exhibit differences both at the first level and within
> nested data types
> * A select * will not cause an exception but using a limit clause will
> * Note also this issue seems to happen only when multiple Drillbit minor
> fragments are involved (concurrency higher than one)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
