[
https://issues.apache.org/jira/browse/DRILL-6223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16420926#comment-16420926
]
salim achouche commented on DRILL-6223:
---------------------------------------
* This PR is only to {color:#FF0000}avoid corruption{color} now that memory
checks are disabled
* As I said, we all agree that Drill enabled an ill-defined functionality
** We have the opportunity to discuss, clarify, formalize it it in a dedicated
JIRA
* Meanwhile, what to do with the current bugs?
** Let's use the following example (which has nothing to do with Schema
Changes but instead is a byproduct of this functionality); assume the following
FS structure
*** ROOT/my_data/T1/\{column-c1}, \{column-c2}, ..
*** ROOT/my_data/T2/\{column-c1}, \{column-c2}, ..
** Assume you issue the following query:
*** SELECT * from dfs.`ROOT/my_data/*`;
** The current code will blindly attempt to read the files thinking they are
originating from the same schema
*** The chance of dangling columns is extremely high
*** What do we do?
**** We can either pretend this is a is schema change issue and try to address
it by inserting compensation logic
**** Avoid corruption by either failing the query or removing dangling columns
**** I chose the latter solution because I don't have clarity on the Schema
Changes functionality
**** It occurred to me that we can also disable Schema Change logic for
SELECT_STAR queries
** To your point about compensation logic in the context of Schema Changes
*** Why do you think it is ok to dynamically include new columns?
*** Yet it is not ok to exclude them?
*** Why did we include a mechanism to report schema changes to the JDBC
client? maybe we thought the consumer app is in a better position to handle
such events; in which case, any compensation logic is unnecessary (?)
* With regards to tests
** I will add a test that triggers this condition
** The test will be deemed successful if there are no runtime failures
** Whether we should add missing columns or not is still being debated and
outside the scope of this JIRA
> Drill fails on Schema changes
> ------------------------------
>
> Key: DRILL-6223
> URL: https://issues.apache.org/jira/browse/DRILL-6223
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Relational Operators
> Affects Versions: 1.10.0, 1.12.0
> Reporter: salim achouche
> Assignee: salim achouche
> Priority: Major
> Fix For: 1.14.0
>
>
> Drill Query Failing when selecting all columns from a Complex Nested Data
> File (Parquet) Set). There are differences in Schema among the files:
> * The Parquet files exhibit differences both at the first level and within
> nested data types
> * A select * will not cause an exception but using a limit clause will
> * Note also this issue seems to happen only when multiple Drillbit minor
> fragments are involved (concurrency higher than one)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
