[ https://issues.apache.org/jira/browse/IMPALA-11342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17569482#comment-17569482 ]
ASF subversion and git services commented on IMPALA-11342: ---------------------------------------------------------- Commit 112f887ea5219e5835d3ca0403faa0ba9ec49115 in impala's branch refs/heads/master from Csaba Ringhofer [ https://gitbox.apache.org/repos/asf?p=impala.git;h=112f887ea ] IMPALA-11342: Fix class loading in Hive UDFs' constructors Loading new classes from the same jar in the constructor of UDFs did not work in the catalog because the URLClassLoader was closed too early. Extended the lifecycle of the class loader a bit to let the catalog finish all initialisation. Note that the instantiation of legacy Hive UDFs doesn't seem necessary in the catalog, we can get all relevant info from the class. Generic UDFs do need to be instantiated to be able to call initialize(). Testing: - added new classes to load in test UDFs and loaded these in constructor / initialize() - ran the Hive UDF ee tests Change-Id: If16e38b8fc3b2577a5d32104ea9e6948b9562e24 Reviewed-on: http://gerrit.cloudera.org:8080/18611 Reviewed-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> > Hive UDFs are unable to use classes from the same Jar in catalogd > ----------------------------------------------------------------- > > Key: IMPALA-11342 > URL: https://issues.apache.org/jira/browse/IMPALA-11342 > Project: IMPALA > Issue Type: Bug > Components: Catalog > Reporter: Csaba Ringhofer > Assignee: Csaba Ringhofer > Priority: Major > > While it is a known limitations that Hive UDFs in Impala cannot load > dependencies from classpath, they should be able to load classes from the > same jar (and thus creating an shaded jar with the dependencies should be a > valid workaround). Meanwhile I got an error when the parent class of the UDF > was a class defined in another package in the jar. > The exception occurred when catalogd tried to instantiate the UDF class: > {code} > I0608 14:50:21.520296 22464 jni-util.cc:286] java.lang.NoClassDefFoundError: > com/esri/core/geometry/Point > at java.lang.Class.getDeclaredConstructors0(Native Method) > at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671) > at java.lang.Class.getConstructor0(Class.java:3075) > at java.lang.Class.getConstructor(Class.java:1825) > at > org.apache.impala.hive.executor.HiveLegacyJavaFunction.instantiateUDFInstance(HiveLegacyJavaFunction.java:115) > at > org.apache.impala.hive.executor.HiveLegacyJavaFunction.<init>(HiveLegacyJavaFunction.java:74) > at > org.apache.impala.hive.executor.HiveJavaFunctionFactoryImpl.create(HiveJavaFunctionFactoryImpl.java:50) > at > org.apache.impala.hive.executor.HiveJavaFunctionFactoryImpl.create(HiveJavaFunctionFactoryImpl.java:67) > at > org.apache.impala.service.CatalogOpExecutor.createFunction(CatalogOpExecutor.java:2117) > at > org.apache.impala.service.CatalogOpExecutor.execDdlRequest(CatalogOpExecutor.java:453) > at org.apache.impala.service.JniCatalog.execDdl(JniCatalog.java:245) > Caused by: java.lang.ClassNotFoundException: com.esri.core.geometry.Point > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > ... 11 more > {code} > The issue seems to be cause by closing the UrlClassLoader too early, after > loading the class but before doing the actual instantiation: > https://github.com/apache/impala/blob/13bbff4e4e5fc5d459cc6f7a5512f84ceba897cd/fe/src/main/java/org/apache/impala/hive/executor/HiveUdfLoader.java#L71 > Note that this is an issue only in the catalogd, not the executors, as the > class loader is kept open longer in that case. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org