Hi Jacek, Thanks for the hints, I would rather have the information statically rather than build a logical plan.
I'm using Apache Calcite to build SQL expressions and then I feed them to spark to run, so the pipeline goes like this: initial query in SQL (from the user) + schema definition (from db) + udf definition (spark + custom lib (Sedona, etc)) => calcite query plan ( + transformations from business logic) => SQL (with Spark Dialect) In my case, udf definitions are an input I would get from whatever is loaded in Spark. That's why it's more convenient to have the information statically here. (bonus: it would be helpful to generate documentation: https://spark.apache.org/docs/latest/api/sql/index.html) For example, this is how you would define acos type signature https://github.com/apache/calcite/blob/5c7be55ffee836366dcc7fefb6adfc0b8c47465f/core/src/main/java/org/apache/calcite/sql/fun/SqlStdOperatorTable.java#L1692-L1696 On Tue, Mar 28, 2023 at 3:27 PM Jacek Laskowski <ja...@japila.pl> wrote: > Hi, > > Interesting question indeed! > > The closest I could get would be to use lookupFunctionBuilder(name: > FunctionIdentifier): Option[FunctionBuilder] [1] followed by extracting the > dataType from T in `type FunctionBuilder = Seq[Expression] => T` which can > be Expression (regular functions) or LogicalPlan (table-valued functions). > Expression has got dataType while LogicalPlan has got output > (or outputAttributes). > > HTH > > Let us know how you're doing. > > BTW, Can you describe how you "using Apache Calcite to run some SQL > transformations on Apache sparks SQL statements"? > > [1] > https://github.com/apache/spark/blob/e60ce3e85081ca8bb247aeceb2681faf6a59a056/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala#L91 > > Pozdrawiam, > Jacek Laskowski > ---- > "The Internals Of" Online Books <https://books.japila.pl/> > Follow me on https://twitter.com/jaceklaskowski > > <https://twitter.com/jaceklaskowski> > > > On Tue, Mar 28, 2023 at 9:01 PM Guillaume Masse < > masse.guilla...@narrative.io> wrote: > >> Hi, >> >> I'm using Apache Calcite to run some SQL transformations on Apache sparks >> SQL statements. I would like to extract the type signature out >> of spark.catalog.listFunctions to be able to register them in Calcite with >> their proper signature. >> >> From the API, I can get the fully qualified class name and the name, but >> unfortunately, the type signature is not present. Would there be a way to >> use reflection to extract? For example: >> >> >> https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala#L424 >> >> Ideally, it would be convenient to get the type signature >> from org.apache.spark.sql.catalog.Function itself when available. >> >> >> -- >> Guillaume Massé >> [Gee-OHM] >> (马赛卫) >> >
