Peter Rozsa has uploaded a new patch set (#25). ( http://gerrit.cloudera.org:8080/19425 )
Change subject: IMPALA-11745: Add Hive's ESRI geospatial functions as builtins ...................................................................... IMPALA-11745: Add Hive's ESRI geospatial functions as builtins This change adds geospatial functions from Hive's ESRI library as builtin UDFs. Plain Hive UDFs are imported without changes, but the generic and varargs functions are handled differently; generic functions are added with all of the combinations of their parameters (cartesian product of the parameters), and varargs functions are unfolded as an nth parameter simple function. The varargs function wrappers are generated at build time and they can be configured in gen_geospatial_udf_wrappers.py. These additional steps are required because of the limitations in Impala's UDF Executor (lack of varargs support and only partial generics support) which could be further improved; in this case, the additional wrapping/mapping steps could be removed. Changes regarding function handling/creating are sourced from https://gerrit.cloudera.org/c/19177 A new backend flag was added to turn this feature on/off as "geospatial_library". The default value is "NONE" which means no geospatial function gets registered as builtin, "HIVE_ESRI" value enables this implementation. The ESRI geospatial implementation for Hive currently only available in Hive 4, but CDP Hive backported it to Hive 3, therefore for Apache Hive this feature is disabled regardless of the "geospatial_library" flag. Known limitations: - ST_MultiLineString, ST_MultiPolygon only works with the WKT overload - ST_Polygon supports a maximum of 6 pairs of coordinates - ST_MultiPoint, ST_LineString supports a maximum of 7 pairs of coordinates - ST_ConvexHull, ST_Union supports a maximum of 6 geoms These limits can be increased in gen_geospatial_udf_wrappers.py Tests: - test_geospatial_udfs.py added based on https://github.com/Esri/spatial-framework-for-hadoop Co-Authored-by: Csaba Ringhofer <csringho...@cloudera.com> Change-Id: If0ca02a70b4ba244778c9db6d14df4423072b225 --- M be/src/common/global-flags.cc M be/src/exprs/hive-udf-call.cc M be/src/util/backend-gflag-util.cc M bin/start-impala-cluster.py M common/function-registry/CMakeLists.txt M common/function-registry/gen_builtins_catalog.py A common/function-registry/gen_geospatial_udf_wrappers.py M common/thrift/BackendGflags.thrift A fe/src/compat-apache-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java A fe/src/compat-hive-3/java/org/apache/impala/compat/HiveEsriGeospatialBuiltins.java M fe/src/main/java/org/apache/impala/catalog/BuiltinsDb.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/ScalarFunction.java A fe/src/main/java/org/apache/impala/hive/executor/BinaryToBinaryHiveLegacyFunctionExtractor.java M fe/src/main/java/org/apache/impala/hive/executor/HiveGenericJavaFunction.java M fe/src/main/java/org/apache/impala/hive/executor/HiveJavaFunction.java M fe/src/main/java/org/apache/impala/hive/executor/HiveJavaFunctionFactory.java M fe/src/main/java/org/apache/impala/hive/executor/HiveJavaFunctionFactoryImpl.java A fe/src/main/java/org/apache/impala/hive/executor/HiveLegacyFunctionExtractor.java M fe/src/main/java/org/apache/impala/hive/executor/HiveLegacyJavaFunction.java M fe/src/main/java/org/apache/impala/hive/executor/ImpalaDoubleWritable.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M fe/src/test/java/org/apache/impala/hive/executor/HiveLegacyJavaFunctionTest.java M fe/src/test/java/org/apache/impala/hive/executor/TestHiveJavaFunctionFactory.java M java/CMakeLists.txt M java/shaded-deps/hive-exec/pom.xml M testdata/datasets/README A testdata/workloads/functional-query/queries/QueryTest/geospatial-esri.test M tests/common/environ.py M tests/common/skip.py A tests/custom_cluster/test_geospatial_library.py A tests/query_test/test_geospatial_functions.py 34 files changed, 3,541 insertions(+), 158 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/19425/25 -- To view, visit http://gerrit.cloudera.org:8080/19425 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If0ca02a70b4ba244778c9db6d14df4423072b225 Gerrit-Change-Number: 19425 Gerrit-PatchSet: 25 Gerrit-Owner: Peter Rozsa <pro...@cloudera.com> Gerrit-Reviewer: Csaba Ringhofer <csringho...@cloudera.com> Gerrit-Reviewer: Daniel Becker <daniel.bec...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Michael Smith <michael.sm...@cloudera.com> Gerrit-Reviewer: Peter Rozsa <pro...@cloudera.com> Gerrit-Reviewer: Steve Carlin <scar...@cloudera.com>
