Re: VectorUDT with spark.ml.linalg.Vector

@Michal Yes, we have public VectorUDT in spark.mllib package at 1.6, and this class is still existing in 2.0. And from 2.0, we provide a new VectorUDT in spark.ml package and make it private temporary (will be public in the near future). Since from 2.0, spark.mllib package will be in maintenance

Re: VectorUDT with spark.ml.linalg.Vector

I'm using Spark 1.6.2 for Vector-based UDAF and this works: def inputSchema: StructType = new StructType().add("input", new VectorUDT()) Maybe it was made private in 2.0 On 17 August 2016 at 05:31, Alexey Svyatkovskiy wrote: > Hi Yanbo, > > Thanks for your reply. I will

Re: VectorUDT with spark.ml.linalg.Vector

Hi Yanbo, Thanks for your reply. I will keep an eye on that pull request. For now, I decided to just put my code inside org.apache.spark.ml to be able to access private classes. Thanks, Alexey On Tue, Aug 16, 2016 at 11:13 PM, Yanbo Liang wrote: > It seams that VectorUDT

Re: VectorUDT with spark.ml.linalg.Vector

It seams that VectorUDT is private and can not be accessed out of Spark currently. It should be public but we need to do some refactor before make it public. You can refer the discussion at https://github.com/apache/spark/pull/12259 . Thanks Yanbo 2016-08-16 9:48 GMT-07:00 alexeys

VectorUDT with spark.ml.linalg.Vector

I am writing an UDAF to be applied to a data frame column of type Vector (spark.ml.linalg.Vector). I rely on spark/ml/linalg so that I do not have to go back and forth between dataframe and RDD. Inside the UDAF, I have to specify a data type for the input, buffer, and output (as usual).