Generic DataType in UDAF

2015-09-25 Thread Ritesh Agrawal
Hi all, I am trying to learn about UDAF and implemented a simple reservoir sample UDAF. It's working fine. However I am not able to figure out what DataType should I use so that its can deal with all DataTypes (simple and complex). For instance currently I have defined my input schema as def

Re: Generic DataType in UDAF

2015-09-25 Thread Yin Huai
Hi Ritesh, Right now, we only allow specific data types defined in the inputSchema. Supporting abstract types (e.g. NumericType) may cause the logic of a UDAF be more complex. It will be great to understand the use cases first. What kinds of possible input data types that you want to support and

Re: Generic DataType in UDAF

2015-09-25 Thread Ritesh Agrawal
hi Yin, I have a written a simple UDAF to generate N samples for each group. I am using reservoir sampling algorithm for this. In this case since the input data type doesn't matter as I am not doing any kind of processing on the input data but just selecting them by random and building an array