Hi fellow Sparkans,
I am building a UDF (in Java) that can return various data types, basically the
signature of the function itself is:
public Object call(String a, Object b, String c, Object d, String e)
throws Exception
When I register my function, I need to provide a type, e.g.:
spark.udf().register("f2", new Udf5(), DataTypes.LongType);
In my test it is a long now, but can become a string or a float. Of course, I
do not know the expected return type before I call the function, which I call
like:
df = df.selectExpr("*", "f2('x1', x, 'c2', y, 'op') as op");
Is there a way to have an Object being returned from a UDF and to store an
Object in a Dataset/dataframe? I don't need to know the datatype at that point
and can leave it hanging for now? Or should I play it safe and always return a
DataTypes.StringType (and then try to transform it if needed)?
I hope I am clear enough :).
Thanks for any tip/idea/comment...
jg
