[
https://issues.apache.org/jira/browse/ORC-1059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Panagiotis Garefalakis updated ORC-1059:
----------------------------------------
Description:
ORC-741 changed the behaviour of column resolution throwing an
IllegalArgumentException when a column is not found in the read schema.
Column resolution is also using by mapSargColumnsToOrcInternalColId when
pushing down filters and throws an IllegalArgumentException exception when a
filter column is not part of the schema instead of ignoring it (as it was the
case in 1.6 and back).
1.6 and back
{code:java}
try {
TypeDescription readerColumn = evolution.getReaderBaseSchema().findSubtype(
columnName, evolution.isSchemaEvolutionCaseAware);
TypeDescription fileColumn = evolution.getFileType(readerColumn);
return fileColumn == null ? -1 : fileColumn.getId();
} catch (IllegalArgumentException e) {
if (LOG.isDebugEnabled()){
LOG.debug("{}", e.getMessage());
}
return -1;
} {code}
1.7 and main
{code:java}
try {
TypeDescription readerColumn = evolution.getReaderBaseSchema().findSubtype(
columnName, evolution.isSchemaEvolutionCaseAware);
return evolution.getFileType(readerColumn);
} catch (IllegalArgumentException e) {
throw new IllegalArgumentException("Filter could not find column with name: "
+
columnName + " on " +
evolution.getReaderBaseSchema(),
e);
} {code}
This makes harder to upgrade to 1.7 on downstream consumers like Hive
https://issues.apache.org/jira/browse/HIVE-25497
was:
ORC-741 changed the behaviour of column resolution failing with an
IllegalArgumentException when a column is not found in the read schema.
Column resolution is also using by mapSargColumnsToOrcInternalColId when
pushing down filters and throws an IllegalArgumentException exception when a
filter column is not part of the schema instead of ignoring it (as it was the
case in 1.6 and back).
1.6 and back
{code:java}
try {
TypeDescription readerColumn = evolution.getReaderBaseSchema().findSubtype(
columnName, evolution.isSchemaEvolutionCaseAware);
TypeDescription fileColumn = evolution.getFileType(readerColumn);
return fileColumn == null ? -1 : fileColumn.getId();
} catch (IllegalArgumentException e) {
if (LOG.isDebugEnabled()){
LOG.debug("{}", e.getMessage());
}
return -1;
} {code}
1.7 and main
{code:java}
try {
TypeDescription readerColumn = evolution.getReaderBaseSchema().findSubtype(
columnName, evolution.isSchemaEvolutionCaseAware);
return evolution.getFileType(readerColumn);
} catch (IllegalArgumentException e) {
throw new IllegalArgumentException("Filter could not find column with name: "
+
columnName + " on " +
evolution.getReaderBaseSchema(),
e);
} {code}
This makes harder to upgrade to 1.7 on downstream consumers like Hive
https://issues.apache.org/jira/browse/HIVE-25497
> Align findColumns behaviour between 1.6 and 1.7 release
> -------------------------------------------------------
>
> Key: ORC-1059
> URL: https://issues.apache.org/jira/browse/ORC-1059
> Project: ORC
> Issue Type: Bug
> Reporter: Panagiotis Garefalakis
> Assignee: Panagiotis Garefalakis
> Priority: Major
>
> ORC-741 changed the behaviour of column resolution throwing an
> IllegalArgumentException when a column is not found in the read schema.
> Column resolution is also using by mapSargColumnsToOrcInternalColId when
> pushing down filters and throws an IllegalArgumentException exception when a
> filter column is not part of the schema instead of ignoring it (as it was the
> case in 1.6 and back).
> 1.6 and back
> {code:java}
> try {
> TypeDescription readerColumn = evolution.getReaderBaseSchema().findSubtype(
> columnName, evolution.isSchemaEvolutionCaseAware);
> TypeDescription fileColumn = evolution.getFileType(readerColumn);
> return fileColumn == null ? -1 : fileColumn.getId();
> } catch (IllegalArgumentException e) {
> if (LOG.isDebugEnabled()){
> LOG.debug("{}", e.getMessage());
> }
> return -1;
> } {code}
> 1.7 and main
> {code:java}
> try {
> TypeDescription readerColumn = evolution.getReaderBaseSchema().findSubtype(
> columnName, evolution.isSchemaEvolutionCaseAware);
> return evolution.getFileType(readerColumn);
> } catch (IllegalArgumentException e) {
> throw new IllegalArgumentException("Filter could not find column with name:
> " +
> columnName + " on " +
> evolution.getReaderBaseSchema(),
> e);
> } {code}
> This makes harder to upgrade to 1.7 on downstream consumers like Hive
> https://issues.apache.org/jira/browse/HIVE-25497
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
