Is it not internal to the Catalyst implementation? I should not be modifying the Spark source to get things to work, do I? :-)
On Wed, Jan 20, 2016 at 12:21 PM, Raghu Ganti <raghuki...@gmail.com> wrote: > Case classes where? > > On Wed, Jan 20, 2016 at 12:21 PM, Andy Grove <andy.gr...@agildata.com> > wrote: > >> Honestly, moving to Scala and using case classes is the path of least >> resistance in the long term. >> >> >> >> Thanks, >> >> Andy. >> >> -- >> >> Andy Grove >> Chief Architect >> AgilData - Simple Streaming SQL that Scales >> www.agildata.com >> >> >> On Wed, Jan 20, 2016 at 10:19 AM, Raghu Ganti <raghuki...@gmail.com> >> wrote: >> >>> Thanks for your reply, Andy. >>> >>> Yes, that is what I concluded based on the Stack trace. The problem is >>> stemming from Java implementation of generics, but I thought this will go >>> away if you compiled against Java 1.8, which solves the issues of proper >>> generic implementation. >>> >>> Any ideas? >>> >>> Also, are you saying that in order for my example to work, I would need >>> to move to Scala and have the UDT implemented in Scala? >>> >>> >>> On Wed, Jan 20, 2016 at 10:27 AM, Andy Grove <andy.gr...@agildata.com> >>> wrote: >>> >>>> Catalyst is expecting a class that implements scala.Row or >>>> scala.Product and is instead finding a Java class. I've run into this issue >>>> a number of times. Dataframe doesn't work so well with Java. Here's a blog >>>> post with more information on this: >>>> >>>> http://www.agildata.com/apache-spark-rdd-vs-dataframe-vs-dataset/ >>>> >>>> >>>> Thanks, >>>> >>>> Andy. >>>> >>>> -- >>>> >>>> Andy Grove >>>> Chief Architect >>>> AgilData - Simple Streaming SQL that Scales >>>> www.agildata.com >>>> >>>> >>>> On Wed, Jan 20, 2016 at 7:07 AM, raghukiran <raghuki...@gmail.com> >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> I created a custom UserDefinedType in Java as follows: >>>>> >>>>> SQLPoint = new UserDefinedType<JavaPoint>() { >>>>> //overriding serialize, deserialize, sqlType, userClass functions here >>>>> } >>>>> >>>>> When creating a dataframe, I am following the manual mapping, I have a >>>>> constructor for JavaPoint - JavaPoint(double x, double y) and a >>>>> Customer >>>>> record as follows: >>>>> >>>>> public class CustomerRecord { >>>>> private int id; >>>>> private String name; >>>>> private Object location; >>>>> >>>>> //setters and getters follow here >>>>> } >>>>> >>>>> Following the example in Spark source, when I create a RDD as follows: >>>>> >>>>> sc.textFile(inputFileName).map(new Function<String, CustomerRecord>() { >>>>> //call method >>>>> CustomerRecord rec = new CustomerRecord(); >>>>> rec.setLocation(SQLPoint.serialize(new JavaPoint(x, y))); >>>>> }); >>>>> >>>>> This results in a MatchError. The stack trace is as follows: >>>>> >>>>> scala.MatchError: [B@45aa3dd5 (of class [B) >>>>> at >>>>> >>>>> org.apache.spark.sql.catalyst.CatalystTypeConverters$StructConverter.toCatalystImpl(CatalystTypeConverters.scala:255) >>>>> at >>>>> >>>>> org.apache.spark.sql.catalyst.CatalystTypeConverters$StructConverter.toCatalystImpl(CatalystTypeConverters.scala:250) >>>>> at >>>>> >>>>> org.apache.spark.sql.catalyst.CatalystTypeConverters$CatalystTypeConverter.toCatalyst(CatalystTypeConverters.scala:102) >>>>> at >>>>> >>>>> org.apache.spark.sql.catalyst.CatalystTypeConverters$$anonfun$createToCatalystConverter$2.apply(CatalystTypeConverters.scala:401) >>>>> at >>>>> >>>>> org.apache.spark.sql.SQLContext$$anonfun$org$apache$spark$sql$SQLContext$$beansToRows$1$$anonfun$apply$1.apply(SQLContext.scala:1358) >>>>> at >>>>> >>>>> org.apache.spark.sql.SQLContext$$anonfun$org$apache$spark$sql$SQLContext$$beansToRows$1$$anonfun$apply$1.apply(SQLContext.scala:1358) >>>>> at >>>>> >>>>> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) >>>>> at >>>>> >>>>> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) >>>>> at >>>>> >>>>> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) >>>>> at >>>>> scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108) >>>>> at >>>>> scala.collection.TraversableLike$class.map(TraversableLike.scala:244) >>>>> at >>>>> scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108) >>>>> at >>>>> >>>>> org.apache.spark.sql.SQLContext$$anonfun$org$apache$spark$sql$SQLContext$$beansToRows$1.apply(SQLContext.scala:1358) >>>>> at >>>>> >>>>> org.apache.spark.sql.SQLContext$$anonfun$org$apache$spark$sql$SQLContext$$beansToRows$1.apply(SQLContext.scala:1356) >>>>> at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) >>>>> at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) >>>>> at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) >>>>> at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) >>>>> at scala.collection.Iterator$$anon$10.next(Iterator.scala:312) >>>>> at scala.collection.Iterator$class.foreach(Iterator.scala:727) >>>>> at >>>>> scala.collection.AbstractIterator.foreach(Iterator.scala:1157) >>>>> at >>>>> scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48) >>>>> at >>>>> >>>>> scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103) >>>>> at >>>>> scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47) >>>>> at scala.collection.TraversableOnce$class.to >>>>> (TraversableOnce.scala:273) >>>>> at scala.collection.AbstractIterator.to(Iterator.scala:1157) >>>>> at >>>>> >>>>> scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265) >>>>> at >>>>> scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157) >>>>> at >>>>> >>>>> scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252) >>>>> at >>>>> scala.collection.AbstractIterator.toArray(Iterator.scala:1157) >>>>> at >>>>> >>>>> org.apache.spark.sql.execution.SparkPlan$$anonfun$5.apply(SparkPlan.scala:212) >>>>> at >>>>> >>>>> org.apache.spark.sql.execution.SparkPlan$$anonfun$5.apply(SparkPlan.scala:212) >>>>> at >>>>> >>>>> org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858) >>>>> at >>>>> >>>>> org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858) >>>>> at >>>>> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) >>>>> at org.apache.spark.scheduler.Task.run(Task.scala:89) >>>>> at >>>>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213) >>>>> at >>>>> >>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>>>> at >>>>> >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>>>> at java.lang.Thread.run(Thread.java:745) >>>>> >>>>> >>>>> >>>>> -- >>>>> View this message in context: >>>>> http://apache-spark-user-list.1001560.n3.nabble.com/Scala-MatchError-in-Spark-SQL-tp26021.html >>>>> Sent from the Apache Spark User List mailing list archive at >>>>> Nabble.com. >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>>>> For additional commands, e-mail: user-h...@spark.apache.org >>>>> >>>>> >>>> >>> >> >
