Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15103 )
Change subject: IMPALA-9324: Correctly handle ORC UNION type in scanner ...................................................................... IMPALA-9324: Correctly handle ORC UNION type in scanner We don't support reading UNION columns. Queries on tables containing UNION types will fail in planning. Error message is metadata loading error. However, scanner may need to read an ORC file with UNION types if the table schema doesn't map to the UNION columns. Though the UNION values won't be read, the scanner need to resolve the file schema, including the UNION types, correctly. In OrcSchemaResolver::BuildSchemaPath, we create a map from ORC type ids to Impala SchemaPath representation for all types of the file. We should deal with UNION types as well. This patch also include some refactor to improve code readability. Tests: - Add tests for table schema and file schema mismatching on all complex types. Change-Id: I452d27b4e281eada00b62ac58af773a3479163ec Reviewed-on: http://gerrit.cloudera.org:8080/15103 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M be/src/exec/hdfs-orc-scanner.cc M be/src/exec/orc-metadata-utils.cc M be/src/exec/orc-metadata-utils.h M common/thrift/generate_error_codes.py M testdata/workloads/functional-query/queries/DataErrorsTest/orc-type-checks.test M tests/query_test/test_scanners.py 6 files changed, 132 insertions(+), 28 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/15103 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I452d27b4e281eada00b62ac58af773a3479163ec Gerrit-Change-Number: 15103 Gerrit-PatchSet: 8 Gerrit-Owner: Quanlong Huang <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Gabor Kaszab <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Norbert Luksa <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
