Hey Guys, I finally found the solution of my weird problem. I’m using OS X and besides the drill installation according to the documentation I also had drill-embedded installed via homebrew (package apache-drill). The homebrew package seemed to interact in a weird way with my „regular“ installation because all the problems (correct log level from logback appeared in the drillbit.log) as well as the UDF problem suddenly disappeared.
Nonetheless, thank you all for your kind help and your efforts! Julian > Am 20.05.2016 um 20:38 schrieb Julian Feinauer <[email protected]>: > > Hey, > > I’m using Drill 1.6.0. > >> Am 20.05.2016 um 20:37 schrieb Abdel Hakim Deneche <[email protected]>: >> >> What version of Drill are you using ? >> >> On Fri, May 20, 2016 at 11:28 AM, Julian Feinauer <[email protected]> >> wrote: >> >>> Hey Abdel, >>> >>> I renamed it due to your previous hint with the static subclass. >>> But the SQL Commands still returns: >>> 0: jdbc:drill:> SELECT myintadd(CAST(position_id AS int),CAST(store_id AS >>> int)) FROM cp.`employee.json`; >>> Error: VALIDATION ERROR: From line 1, column 8 to line 1, column 63: No >>> match found for function signature myintadd(<NUMERIC>, <NUMERIC>) >>> >>> So this has to be something wrong with my drill setup? >>> >>> All I do is to start my drillbit with drillbit.sh start (or restart) and >>> then start drill-conf where the conf/drill-override.conf contains: >>> drill.exec: { >>> cluster-id: "drillbits1", >>> zk.connect: "localhost:2181" >>> } >>> >>> There’s one other thing that seems strange to me with my setup: >>> Although I set the loglevel in conf/logback.xml to trace the drillbit.log >>> file does not show any trace outputs. >>> >>> Am I doing something wrong with my setup? >>> >>> Thanks >>> Julian >>> >>>> Am 20.05.2016 um 20:13 schrieb Abdel Hakim Deneche < >>> [email protected]>: >>>> >>>> You defined your function as myintadd: >>>> >>>> >>>> @FunctionTemplate(name = "*myintadd*", scope ... >>>> >>>> >>>> The following query worked fine for me: >>>> >>>> SELECT *myintadd*(CAST(position_id AS int),CAST(store_id AS int)) FROM >>>>> cp.`employee.json`; >>>> >>>> >>>> Thanks >>>> >>>> >>>> >>>> >>>> >>>> On Fri, May 20, 2016 at 10:17 AM, Julian Feinauer < >>> [email protected]> >>>> wrote: >>>> >>>>> Dear Jacques, >>>>> Dear Abdel, >>>>> >>>>> thanks for your kind help. >>>>> >>>>> jar tf drillUDF-1.0-SNAPSHOT.jar returns: >>>>> >>>>> META-INF/ >>>>> META-INF/MANIFEST.MF >>>>> org/ >>>>> org/apache/ >>>>> org/apache/drill/ >>>>> org/apache/drill/contrib/ >>>>> org/apache/drill/contrib/function/ >>>>> org/julian/ >>>>> drill-module.conf >>>>> IntIntAdd.class >>>>> org/apache/drill/contrib/function/IntIntAdd.class >>>>> org/julian/IntIntAdd$myintadd.class >>>>> org/julian/IntIntAdd.class >>>>> org/julian/IntIntAdd.java >>>>> META-INF/maven/ >>>>> META-INF/maven/julian.test/ >>>>> META-INF/maven/julian.test/drillUDF/ >>>>> META-INF/maven/julian.test/drillUDF/pom.xml >>>>> META-INF/maven/julian.test/drillUDF/pom.properties >>>>> >>>>> And jar tf drillUDF-1.0-SNAPSHOT-sources.jar returns >>>>> META-INF/ >>>>> META-INF/MANIFEST.MF >>>>> org/ >>>>> org/apache/ >>>>> org/apache/drill/ >>>>> org/apache/drill/contrib/ >>>>> org/apache/drill/contrib/function/ >>>>> org/julian/ >>>>> org/julian/IntIntAdd.java >>>>> drill-module.conf >>>>> >>>>> so the drill-module.conf is in both roots. >>>>> >>>>> @Abdel: >>>>> How or where can I share the jar files with you? >>>>> >>>>> Best regards >>>>> Julian >>>>> >>>>>> Am 20.05.2016 um 17:31 schrieb Jacques Nadeau <[email protected]>: >>>>>> >>>>>> Can you run jar tf myudf.jar against your jar files? Since Drill is >>> not >>>>>> detecting the jar file, we need to resolve that first. The >>>>>> drill-module.conf must be in the root of each jar file that should be >>>>>> included. Lets start by verifying that. >>>>>> >>>>>> -- >>>>>> Jacques Nadeau >>>>>> CTO and Co-Founder, Dremio >>>>>> >>>>>> On Fri, May 20, 2016 at 6:08 AM, Julian Feinauer < >>> [email protected] >>>>>> >>>>>> wrote: >>>>>> >>>>>>> Dear all, >>>>>>> >>>>>>> thank you very much for all your replies. >>>>>>> I tried everything but it is still not working. >>>>>>> >>>>>>> 1. I copy both files (classes and sources) in the /jars/3rdparty >>>>> directory >>>>>>> 2. I restarted the drillbit after this (I use only one drillbit and >>>>>>> drill-conf both running on my local machine) >>>>>>> 3. I changed the class to a static subclass >>>>>>> 4. I have the drill-module.conf in my ressources >>>>>>> 4. The error appears in the drillbit.log as soon as i call the udf >>>>> because >>>>>>> it is not recognized by drill. >>>>>>> On startup drillbit.log states all the packages and jars that are >>>>> scanned >>>>>>> and my custom jar is not listed there. >>>>>>> Therefore I think it is a problem with the class loader or something >>>>>>> related? >>>>>>> >>>>>>> Could this be possible? >>>>>>> >>>>>>> Greetings >>>>>>> Julian >>>>>>> >>>>>>>> Am 20.05.2016 um 14:36 schrieb Tugdual Grall <[email protected]>: >>>>>>>> >>>>>>>> Hi >>>>>>>> >>>>>>>> Be sure you deploy on each nodes, 2 jars: >>>>>>>> - the jar containing the classes >>>>>>>> - the jar contains the sources >>>>>>>> >>>>>>>> The POM.xml in the simple function examples contains the maven >>>>>>>> configuration to generate these 2 files, be sure you have the same in >>>>>>> your >>>>>>>> project: >>>>>>>> >>>>>>> >>>>> >>> https://github.com/tgrall/drill-simple-mask-function/blob/master/pom.xml#L24-L37 >>>>>>>> >>>>>>>> and you have restarted the drillbit >>>>>>>> >>>>>>>> Regards >>>>>>>> Tug >>>>>>>> @tgrall >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Fri, May 20, 2016 at 12:34 AM, Abdel Hakim Deneche < >>>>>>> [email protected] >>>>>>>>> wrote: >>>>>>>> >>>>>>>>> the example I gave you was incomplete, here is what I meant to send: >>>>>>>>> >>>>>>>>> public class MyUDF { >>>>>>>>> >>>>>>>>> @FunctionTemplate(name = „myaddints", scope = FunctionTemplate. >>>>>>>>> FunctionScope.SIMPLE, nulls = >>>>>>>>> FunctionTemplate.NullHandling.NULL_IF_NULL) >>>>>>>>> public *static *class IntIntAdd implements DrillSimpleFunc { >>>>>>>>> ... >>>>>>>>> } >>>>>>>>> >>>>>>>>> } >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, May 19, 2016 at 3:33 PM, Abdel Hakim Deneche < >>>>>>>>> [email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hey Julian, >>>>>>>>>> >>>>>>>>>> one more thing you could try out: declare the UDF as a static class >>>>>>>>> inside >>>>>>>>>> another class: >>>>>>>>>> >>>>>>>>>> public class MyUDF { >>>>>>>>>> >>>>>>>>>> @FunctionTemplate(name = „myaddints", scope = FunctionTemplate. >>>>>>>>>> FunctionScope.SIMPLE, nulls = FunctionTemplate.NullHandling. >>>>>>>>>> NULL_IF_NULL) >>>>>>>>>> public class IntIntAdd implements DrillSimpleFunc { >>>>>>>>>> ... >>>>>>>>>> } >>>>>>>>>> >>>>>>>>>> } >>>>>>>>>> >>>>>>>>>> Take a look at the following page to see an examples of UDFs: >>>>>>>>>> http://drill.apache.org/docs/custom-function-interfaces/ >>>>>>>>>> >>>>>>>>>> If this doesn't work check the drillbit log, it should print an >>> error >>>>>>>>>> message when it's starting up if something's wrong with your UDF. >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Thu, May 19, 2016 at 3:31 AM, Julian Feinauer < >>>>>>> [email protected] >>>>>>>>>> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Dear folks, >>>>>>>>>>> >>>>>>>>>>> I’m currently experimenting with user defined functions in drill >>> but >>>>>>> I’m >>>>>>>>>>> not able to get them to work on my drillbits. >>>>>>>>>>> I always get the error: Error: VALIDATION ERROR: From line 1, >>>>> column 8 >>>>>>>>> to >>>>>>>>>>> line 1, column 41: No match found for function signature >>>>>>>>> myaddints(<ANY>, >>>>>>>>>>> <ANY>). >>>>>>>>>>> >>>>>>>>>>> I already went through all the tips I found in the mailing list. >>>>>>>>>>> The jar contains a drill-module.conf with the content: >>>>>>>>>>> drill.classpath.scanning.packages += "org.julian" >>>>>>>>>>> And the UDF is defined as: >>>>>>>>>>> package org.julian; >>>>>>>>>>> >>>>>>>>>>> import ... >>>>>>>>>>> >>>>>>>>>>> @FunctionTemplate(name = „myaddints", scope = >>>>>>>>>>> FunctionTemplate.FunctionScope.SIMPLE, nulls = >>>>>>>>>>> FunctionTemplate.NullHandling.NULL_IF_NULL) >>>>>>>>>>> public class IntIntAdd implements DrillSimpleFunc { >>>>>>>>>>> >>>>>>>>>>> @Param >>>>>>>>>>> IntHolder in1; >>>>>>>>>>> >>>>>>>>>>> @Param >>>>>>>>>>> IntHolder in2; >>>>>>>>>>> >>>>>>>>>>> @Output >>>>>>>>>>> IntHolder out; >>>>>>>>>>> >>>>>>>>>>> public void setup() {}; >>>>>>>>>>> >>>>>>>>>>> public void eval() { >>>>>>>>>>> out.value = in1.value + in2.value; >>>>>>>>>>> } >>>>>>>>>>> >>>>>>>>>>> } >>>>>>>>>>> I tried to find the bug using the logs and encountered the >>> following >>>>>>>>>>> messages from the drillbit.log: >>>>>>>>>>> Base Configuration: >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>> >>>>>>> >>>>> >>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-common-1.6.0.jar!/drill-default.conf >>>>>>>>>>> >>>>>>>>>>> Intermediate Configuration and Plugin files, in order of >>> precedence: >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>> >>>>>>> >>>>> >>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-gis-1.6.0.jar!/drill-module.conf >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>> >>>>>>> >>>>> >>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-logical-1.6.0.jar!/drill-module.conf >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>> >>>>>>> >>>>> >>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-kudu-storage-1.6.0.jar!/drill-module.conf >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>> >>>>>>> >>>>> >>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/3rdparty/drill-memory-base-1.6.0.jar!/drill-module.conf >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>> >>>>>>> >>>>> >>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-hive-exec-shaded-1.6.0.jar!/drill-module.conf >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>> >>>>>>> >>>>> >>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-memory-base-1.6.0.jar!/drill-module.conf >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>> >>>>>>> >>>>> >>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-mongo-storage-1.6.0.jar!/drill-module.conf >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>> >>>>>>> >>>>> >>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-java-exec-1.6.0.jar!/drill-module.conf >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>> >>>>>>> >>>>> >>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-common-1.6.0.jar!/drill-module.conf >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>> >>>>>>> >>>>> >>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-storage-hbase-1.6.0.jar!/drill-module.conf >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>> >>>>>>> >>>>> >>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-jdbc-storage-1.6.0.jar!/drill-module.conf >>>>>>>>>>> - >>>>>>>>>>> >>>>>>>>> >>>>>>> >>>>> >>> jar:file:/usr/local/Cellar/apache-drill/1.6.0/libexec/jars/drill-storage-hive-core-1.6.0.jar!/drill-module.conf >>>>>>>>>>> >>>>>>>>>>> which let’s me think that my jar file (drillUDF-1.0-SNAPSHOT.jar) >>> is >>>>>>> not >>>>>>>>>>> scanned altough it is located in jars/3rdparty. >>>>>>>>>>> >>>>>>>>>>> Am I doing something wrong or am I missing something or has >>> anyone a >>>>>>>>> good >>>>>>>>>>> hint for me? >>>>>>>>>>> >>>>>>>>>>> Thank you already! >>>>>>>>>>> Julian >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> >>>>>>>>>> Abdelhakim Deneche >>>>>>>>>> >>>>>>>>>> Software Engineer >>>>>>>>>> >>>>>>>>>> <http://www.mapr.com/> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Now Available - Free Hadoop On-Demand Training >>>>>>>>>> < >>>>>>>>> >>>>>>> >>>>> >>> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> Abdelhakim Deneche >>>>>>>>> >>>>>>>>> Software Engineer >>>>>>>>> >>>>>>>>> <http://www.mapr.com/> >>>>>>>>> >>>>>>>>> >>>>>>>>> Now Available - Free Hadoop On-Demand Training >>>>>>>>> < >>>>>>>>> >>>>>>> >>>>> >>> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available >>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> >>>> Abdelhakim Deneche >>>> >>>> Software Engineer >>>> >>>> <http://www.mapr.com/> >>>> >>>> >>>> Now Available - Free Hadoop On-Demand Training >>>> < >>> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available >>>> >>> >>> >> >> >> -- >> >> Abdelhakim Deneche >> >> Software Engineer >> >> <http://www.mapr.com/> >> >> >> Now Available - Free Hadoop On-Demand Training >> <http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available> >
