Using Hive UDTF in SparkSQL

[Lokesh Yadav]

Hello

I am trying to use a Hive UDTF function in spark SQL. But somehow its not
working for me as intended and I am not able to understand the behavior.

When I try to register a function like this:
create temporary function SampleUDTF_01 as
'com.fl.experiments.sparkHive.SampleUDTF' using JAR
'hdfs:///user/root/sparkHive-1.0.0.jar';
It successfully registers the function, but gives me a 'not a registered
function' error when I try to run that function. Also it doesn't show up in
the list when I do a 'show functions'.

Another case:
When I try to register the same function as a temporary function using a
local jar (the hdfs path doesn't work with temporary function, that is
weird too), it registers, and I am able to successfully run that function
as well. Another weird thing is that I am not able to drop that function
using the 'drop function ...' statement. This the functions shows up in the
function registry.

I am stuck with this, any help would be really appreciated.
Thanks

Regards,
Lokesh Yadav

How to Register Permanent User-Defined-Functions (UDFs) in SparkSQL

[Lokesh Yadav]

Hi
with sqlContext we can register a UDF like
this: sqlContext.udf.register("sample_fn", sample_fn _ )
But this UDF is limited to that particular sqlContext only. I wish to make
the registration persistent, so that I can access the same UDF in any
subsequent sqlcontext.
Or is there any other way to register UDFs in sparkSQL so that they remain
persistent?

Regards
Lokesh