[
https://issues.apache.org/jira/browse/DRILL-6223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16422069#comment-16422069
]
Parth Chandra commented on DRILL-6223:
--------------------------------------
{quote}Sorry to say, I still disagree with this statement: "This pull request
adds logic to detect and eliminate dangling columns".
{quote}
Adding some more background -
[~paul-rogers] has this exactly right. At the high level, dropping any columns
for star queries is not the solution as a star queries explicitly asks for all
columns.
More importantly, as Paul pointed out, the solution (in some cases) is to
provide a schema.
For Parquet (or any other source that has schema information), we might solve
the problem by creating a union of the columns (and also take care of type
changes by promoting columns to a common type). Parquet currently does not do
that. Worse, it overrides the schema seen so far with the new schema when a
change is encountered (I would suggest, once again, a perusal of the Parquet
metadata cache code). This means in the presence of schema change with Parquet
files, even the planning gets off on the wrong foot.
The only solution, again, as Paul mentioned, is to provide a composite schema
by inferring it or asking the user to provide one. The latter is hard because
sometimes the user does not have the schema or because it is mutating all the
time (it happens when people move fast and break things). This is exactly when
they want to use Drill, and, in fact, this was a primary use case for the early
design of Drill. As a design constraint, we had to assume that we could not
know the schema until runtime (we may have taken it too far :( ).
We could try to infer the schema by doing a complete scan but that has the same
issues as asking the user to specify as schema and, in addition, can take
really really long.
> Drill fails on Schema changes
> ------------------------------
>
> Key: DRILL-6223
> URL: https://issues.apache.org/jira/browse/DRILL-6223
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Relational Operators
> Affects Versions: 1.10.0, 1.12.0
> Reporter: salim achouche
> Assignee: salim achouche
> Priority: Major
> Fix For: 1.14.0
>
>
> Drill Query Failing when selecting all columns from a Complex Nested Data
> File (Parquet) Set). There are differences in Schema among the files:
> * The Parquet files exhibit differences both at the first level and within
> nested data types
> * A select * will not cause an exception but using a limit clause will
> * Note also this issue seems to happen only when multiple Drillbit minor
> fragments are involved (concurrency higher than one)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
