Question on Dynamic UDF with hdfs

[François Méthot]

Hi,

  Dynamic UDF is very neat new feature. We are trying to make it work on
HDFS.


we are using a config that looks like this:

drill.exec.udf {
  retry-attempts: 5,
  directory : {
    fs: "hdfs:/ourname.node:8020",
    root: "/an_hdfs/drill_path/udf",
    staging: "/staging",
    registry: "/registry",
    tmp: "/a_drillbit/local/tmp"
    local: "/udf"
}
}

We drop UDF jar in staging directory on hdfs.

>hadoop fs -copyFromLocal drill_test.udf-1.0.0.jar
/an_hdfs/drill_path/udf/staging/drill.test_udf.jar

Then in Drill :

CREATE FUNCTION USING JAR 'drill.test_udf.jar';

1st problem:
  It returns:
      Files does not exist:
/an_hdfs/drill_path/udf/drill.test_udf-sources.jar


So we copy the same file again (with added "-sources"):
>hadoop fs -copyFromLocal drill_test.udf-1.0.0.jar
/an_hdfs/drill_path/udf/drill.test_udf-sources.jar

So that we have 2 identical file, one has "-sources"


Redo the create function:
CREATE FUNCTION USING JAR 'drill.test_udf.jar';
This time it works:
   The following  UDFs in jar drill.test_udf.jar have been registered:
   [hello_word(VARCHAR_OPTIONAL), hello_world(VACHAR-REQUIRED)]

   ( Note:  if we do CREATE FUNCTION USING JAR
'drill.test_udf-sources.jar', it complains it can't find
'drill.test_udf-sources-sources.jar')

2nd problem
  We are unable to use the UDF,
  No Match for function signature hello_world(<ANY>)...

Before I debug the function code it self, I would like to make sure that
the UDF is actually  being seen locally by each drillbit.

Based on the doc for "Local" property:
   "The relative path concatenated to the Drill temporary directory to
indicate the local UDF directory. "

I was expecting drill to copy the Dynamic UDF jar from hdfs to a local dir
of each drill bit in
/a_drillbit/local/tmp/udf

Is it where it should be, based on our config?

Thanks