Attila Jeges has uploaded a new patch set (#20). ( http://gerrit.cloudera.org:8080/12481 )
Change subject: IMPALA-7368: Add initial support for DATE type ...................................................................... IMPALA-7368: Add initial support for DATE type DATE values describe a particular year/month/day in the form yyyy-MM-dd. For example: DATE '2019-02-15'. DATE values do not have a time of day component. The range of values supported for the DATE type is 0000-01-01 to 9999-12-31. This initial DATE type support covers TEXT and HBASE fileformats only. 'DateValue' is used as the internal type to represent DATE values. The changes are as follows: - Support for DATE literal syntax. - Explicit casting between DATE and other types (note that invalid casts will fail with an error, just like invalid DECIMAL_V2 casts): - from STRING to DATE. The string value must be formatted as yyyy-MM-dd HH:mm:ss.SSSSSSSSS. The date component is mandatory, the time component is optional. If the time component is present, it will be truncated silently. - from DATE to STRING. The resulting string value is formatted as yyyy-MM-dd. - from TIMESTAMP to DATE. The source timestamp's time of day component is ignored. - from DATE to TIMESTAMP. The target timestamp's time of day component is set to 00:00:00. - Implicit casting between DATE and other types: - from STRING to DATE if the source string value is used in a context where a DATE value is expected. - from DATE to TIMESTAMP if the source date value is used in a context where a TIMESTAMP value is expected. - Since STRING -> DATE, STRING -> TIMESTAMP and DATE -> TIMESTAMP implicit conversions are now all possible, the existing function overload resolution logic is not adequate anymore. For example, it resolves the if(false, '2011-01-01', DATE '1499-02-02') function call to the if(BOOLEAN, TIMESTAMP, TIMESTAMP) version of the overloaded function, instead of the if(BOOLEAN, DATE, DATE) version. This is clearly wrong, so the function overload resolution logic had to be changed to resolve function calls to the best-fit overloaded function definition if there are multiple applicable candidates. An overloaded function definition is an applicable candidate for a function call if each actual parameter in the function call either matches the corresponding formal parameter's type (without casting) or is implicitly castable to that type. When looking for the best-fit applicable candidate, a parameter match score (i.e. the number of actual parameters in the function call that match their corresponding formal parameter's type without casting) is calculated and the applicable candidate with the highest parameter match score is chosen. There's one more issue that the new resolution logic has to address: if two applicable candidates have the same parameter match score and the only difference between the two is that the first one requires a STRING -> TIMESTAMP implicit cast for some of its parameters while the second one requires a STRING -> DATE implicit cast for the same parameters then the first candidate has to be chosen not to break backward compatibility. E.g: year('2019-02-15') function call must resolve to year(TIMESTAMP) instead of year(DATE). Note, that year(DATE) is not implemented yet, so this is not an issue at the moment but it will be in the future. When the resolution algorithm considers overloaded function definitions, first it orders them lexicographically by the types in their parameter lists. To ensure the backward compatible behavior Primitivetype.DATE enum value has to come after PrimitiveType.TIMESTAMP. - Codegen infrastructure changes for expression evaluation. - 'IS [NOT] NULL' and '[NOT] IN' predicates. - Common comparison operators (including the 'BETWEEN' operator). - Infrastructure changes for built-in functions. - Some built-in functions: conditional, aggregate, analytical and math functions. - C++ UDF/UDA support. - Support partitioning and grouping by DATE. - Beeswax, HiveServer2 support. These items are tightly coupled and it makes sense to implement them in one change-set. Testing: - A new partitioned TEXT table 'functional.date_tbl' (and the corresponding HBASE table 'functional_hbase.date_tbl') was introduced for DATE-related tests. - BE and FE tests were extended to cover DATE type. - E2E tests: - since DATE type is supported for TEXT and HBASE fileformats only, most DATE tests were implemented separately in tests/query_test/test_date_queries.py. Note, that this change-set is not a complete DATE type implementation, but it lays the foundation for future work: - Add date support to the random query generator. - Implement a complete set of built-in functions. - Add Parquet support. - Add Kudu support. - Optionally support Avro and ORC. For further details, see IMPALA-6169. Change-Id: Iea8155ef09557e0afa2f8b2d0b2dc9d0896dc30f --- M be/src/codegen/codegen-anyval.cc M be/src/codegen/codegen-anyval.h M be/src/codegen/gen_ir_descriptions.py M be/src/codegen/llvm-codegen.cc M be/src/exec/aggregator.cc M be/src/exec/data-source-scan-node.cc M be/src/exec/hash-table.cc M be/src/exec/hdfs-scanner-ir.cc M be/src/exec/hdfs-table-sink.cc M be/src/exec/text-converter.cc M be/src/exec/text-converter.inline.h M be/src/exprs/agg-fn-evaluator.cc M be/src/exprs/aggregate-functions-ir.cc M be/src/exprs/aggregate-functions.h M be/src/exprs/anyval-util.cc M be/src/exprs/anyval-util.h M be/src/exprs/case-expr.cc M be/src/exprs/case-expr.h M be/src/exprs/cast-functions-ir.cc M be/src/exprs/cast-functions.h M be/src/exprs/conditional-functions-ir.cc M be/src/exprs/conditional-functions.h M be/src/exprs/expr-test.cc M be/src/exprs/expr-value.h M be/src/exprs/hive-udf-call.cc M be/src/exprs/hive-udf-call.h M be/src/exprs/in-predicate-ir.cc M be/src/exprs/in-predicate.h M be/src/exprs/is-null-predicate-ir.cc M be/src/exprs/literal.cc M be/src/exprs/literal.h M be/src/exprs/math-functions-ir.cc M be/src/exprs/math-functions.h M be/src/exprs/null-literal.cc M be/src/exprs/null-literal.h M be/src/exprs/operators-ir.cc M be/src/exprs/operators.h M be/src/exprs/scalar-expr-evaluator.cc M be/src/exprs/scalar-expr-evaluator.h M be/src/exprs/scalar-expr-ir.cc M be/src/exprs/scalar-expr.cc M be/src/exprs/scalar-expr.h M be/src/exprs/scalar-fn-call.cc M be/src/exprs/scalar-fn-call.h M be/src/exprs/slot-ref.cc M be/src/exprs/slot-ref.h M be/src/exprs/timestamp-functions.h M be/src/exprs/utility-functions-ir.cc M be/src/exprs/utility-functions.h M be/src/rpc/thrift-util.cc M be/src/runtime/date-parse-util.cc M be/src/runtime/date-parse-util.h M be/src/runtime/date-test.cc M be/src/runtime/date-value.cc M be/src/runtime/date-value.h M be/src/runtime/datetime-parse-util.cc M be/src/runtime/datetime-parse-util.h M be/src/runtime/raw-value-ir.cc M be/src/runtime/raw-value-test.cc M be/src/runtime/raw-value.cc M be/src/runtime/raw-value.inline.h M be/src/runtime/timestamp-parse-util.cc M be/src/runtime/timestamp-value.h M be/src/runtime/timestamp-value.inline.h M be/src/runtime/types.cc M be/src/runtime/types.h M be/src/service/fe-support.cc M be/src/service/hs2-util.cc M be/src/service/query-result-set.cc M be/src/testutil/test-udas.cc M be/src/testutil/test-udfs.cc M be/src/udf/udf-test.cc M be/src/udf/udf.h M be/src/util/CMakeLists.txt M be/src/util/static-asserts.cc M be/src/util/string-parser-test.cc M be/src/util/string-parser.h M be/src/util/symbols-util-test.cc M be/src/util/symbols-util.cc M bin/rat_exclude_files.txt M common/function-registry/impala_functions.py M common/thrift/Exprs.thrift M common/thrift/ImpalaInternalService.thrift M ext-data-source/test/src/main/java/org/apache/impala/extdatasource/AllTypesDataSource.java M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/CastExpr.java A fe/src/main/java/org/apache/impala/analysis/DateLiteral.java M fe/src/main/java/org/apache/impala/analysis/LiteralExpr.java M fe/src/main/java/org/apache/impala/analysis/PartitionSpec.java M fe/src/main/java/org/apache/impala/analysis/StringLiteral.java M fe/src/main/java/org/apache/impala/catalog/BuiltinsDb.java M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/ColumnStats.java M fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java M fe/src/main/java/org/apache/impala/catalog/Function.java M fe/src/main/java/org/apache/impala/catalog/PrimitiveType.java M fe/src/main/java/org/apache/impala/catalog/ScalarFunction.java M fe/src/main/java/org/apache/impala/catalog/ScalarType.java M fe/src/main/java/org/apache/impala/catalog/Type.java M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/main/java/org/apache/impala/service/FeSupport.java M fe/src/main/java/org/apache/impala/util/AvroSchemaConverter.java M fe/src/main/java/org/apache/impala/util/AvroSchemaParser.java M fe/src/main/java/org/apache/impala/util/FunctionUtils.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeExprsTest.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeSubqueriesTest.java M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java M fe/src/test/java/org/apache/impala/analysis/ExprRewriteRulesTest.java M fe/src/test/java/org/apache/impala/analysis/ExprRewriterTest.java M fe/src/test/java/org/apache/impala/analysis/LiteralExprTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java M fe/src/test/java/org/apache/impala/catalog/PartialCatalogInfoTest.java M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M fe/src/test/java/org/apache/impala/hive/executor/TestUdf.java M fe/src/test/java/org/apache/impala/service/JdbcTest.java M testdata/bin/compute-table-stats.sh M testdata/bin/create-data-source-table.sql M testdata/data/README A testdata/data/date_tbl.avro A testdata/data/date_tbl.parquet A testdata/data/date_tbl/0000.txt A testdata/data/date_tbl/0001.txt A testdata/data/date_tbl/0002.txt A testdata/data/date_tbl/0003.txt A testdata/data/date_tbl_error/0000.txt A testdata/data/date_tbl_error/0001.txt A testdata/data/date_tbl_error/0002.txt A testdata/data/date_tbl_error/0003.txt M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv M testdata/workloads/functional-planner/queries/PlannerTest/constant-folding.test M testdata/workloads/functional-planner/queries/PlannerTest/data-source-tables.test M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test M testdata/workloads/functional-query/queries/DataErrorsTest/hdfs-scan-node-errors.test M testdata/workloads/functional-query/queries/QueryTest/analytic-fns.test A testdata/workloads/functional-query/queries/QueryTest/compute-stats-date.test M testdata/workloads/functional-query/queries/QueryTest/data-source-tables.test A testdata/workloads/functional-query/queries/QueryTest/date-partitioning.test A testdata/workloads/functional-query/queries/QueryTest/date-text-only-support.test A testdata/workloads/functional-query/queries/QueryTest/date.test M testdata/workloads/functional-query/queries/QueryTest/decimal-exprs.test M testdata/workloads/functional-query/queries/QueryTest/hbase-compute-stats-incremental.test M testdata/workloads/functional-query/queries/QueryTest/hbase-compute-stats.test M testdata/workloads/functional-query/queries/QueryTest/hbase-inserts.test M testdata/workloads/functional-query/queries/QueryTest/hbase-scan-node.test M testdata/workloads/functional-query/queries/QueryTest/joins.test M testdata/workloads/functional-query/queries/QueryTest/misc.test M testdata/workloads/functional-query/queries/QueryTest/subquery.test M testdata/workloads/functional-query/queries/QueryTest/uda.test M testdata/workloads/functional-query/queries/QueryTest/udf.test M testdata/workloads/functional-query/queries/QueryTest/views-ddl.test M tests/custom_cluster/test_permanent_udfs.py M tests/hs2/test_fetch.py M tests/metadata/test_compute_stats.py A tests/query_test/test_date_queries.py M tests/query_test/test_udfs.py M tests/test-hive-udfs/src/main/java/org/apache/impala/TestUdf.java 163 files changed, 4,260 insertions(+), 474 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/81/12481/20 -- To view, visit http://gerrit.cloudera.org:8080/12481 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iea8155ef09557e0afa2f8b2d0b2dc9d0896dc30f Gerrit-Change-Number: 12481 Gerrit-PatchSet: 20 Gerrit-Owner: Attila Jeges <atti...@cloudera.com> Gerrit-Reviewer: Attila Jeges <atti...@cloudera.com> Gerrit-Reviewer: Csaba Ringhofer <csringho...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Lars Volker <l...@cloudera.com> Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com> Gerrit-Reviewer: Todd Lipcon <t...@apache.org>