Case classes where? On Wed, Jan 20, 2016 at 12:21 PM, Andy Grove <andy.gr...@agildata.com> wrote:
> Honestly, moving to Scala and using case classes is the path of least > resistance in the long term. > > > > Thanks, > > Andy. > > -- > > Andy Grove > Chief Architect > AgilData - Simple Streaming SQL that Scales > www.agildata.com > > > On Wed, Jan 20, 2016 at 10:19 AM, Raghu Ganti <raghuki...@gmail.com> > wrote: > >> Thanks for your reply, Andy. >> >> Yes, that is what I concluded based on the Stack trace. The problem is >> stemming from Java implementation of generics, but I thought this will go >> away if you compiled against Java 1.8, which solves the issues of proper >> generic implementation. >> >> Any ideas? >> >> Also, are you saying that in order for my example to work, I would need >> to move to Scala and have the UDT implemented in Scala? >> >> >> On Wed, Jan 20, 2016 at 10:27 AM, Andy Grove <andy.gr...@agildata.com> >> wrote: >> >>> Catalyst is expecting a class that implements scala.Row or scala.Product >>> and is instead finding a Java class. I've run into this issue a number of >>> times. Dataframe doesn't work so well with Java. Here's a blog post with >>> more information on this: >>> >>> http://www.agildata.com/apache-spark-rdd-vs-dataframe-vs-dataset/ >>> >>> >>> Thanks, >>> >>> Andy. >>> >>> -- >>> >>> Andy Grove >>> Chief Architect >>> AgilData - Simple Streaming SQL that Scales >>> www.agildata.com >>> >>> >>> On Wed, Jan 20, 2016 at 7:07 AM, raghukiran <raghuki...@gmail.com> >>> wrote: >>> >>>> Hi, >>>> >>>> I created a custom UserDefinedType in Java as follows: >>>> >>>> SQLPoint = new UserDefinedType<JavaPoint>() { >>>> //overriding serialize, deserialize, sqlType, userClass functions here >>>> } >>>> >>>> When creating a dataframe, I am following the manual mapping, I have a >>>> constructor for JavaPoint - JavaPoint(double x, double y) and a Customer >>>> record as follows: >>>> >>>> public class CustomerRecord { >>>> private int id; >>>> private String name; >>>> private Object location; >>>> >>>> //setters and getters follow here >>>> } >>>> >>>> Following the example in Spark source, when I create a RDD as follows: >>>> >>>> sc.textFile(inputFileName).map(new Function<String, CustomerRecord>() { >>>> //call method >>>> CustomerRecord rec = new CustomerRecord(); >>>> rec.setLocation(SQLPoint.serialize(new JavaPoint(x, y))); >>>> }); >>>> >>>> This results in a MatchError. The stack trace is as follows: >>>> >>>> scala.MatchError: [B@45aa3dd5 (of class [B) >>>> at >>>> >>>> org.apache.spark.sql.catalyst.CatalystTypeConverters$StructConverter.toCatalystImpl(CatalystTypeConverters.scala:255) >>>> at >>>> >>>> org.apache.spark.sql.catalyst.CatalystTypeConverters$StructConverter.toCatalystImpl(CatalystTypeConverters.scala:250) >>>> at >>>> >>>> org.apache.spark.sql.catalyst.CatalystTypeConverters$CatalystTypeConverter.toCatalyst(CatalystTypeConverters.scala:102) >>>> at >>>> >>>> org.apache.spark.sql.catalyst.CatalystTypeConverters$$anonfun$createToCatalystConverter$2.apply(CatalystTypeConverters.scala:401) >>>> at >>>> >>>> org.apache.spark.sql.SQLContext$$anonfun$org$apache$spark$sql$SQLContext$$beansToRows$1$$anonfun$apply$1.apply(SQLContext.scala:1358) >>>> at >>>> >>>> org.apache.spark.sql.SQLContext$$anonfun$org$apache$spark$sql$SQLContext$$beansToRows$1$$anonfun$apply$1.apply(SQLContext.scala:1358) >>>> at >>>> >>>> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) >>>> at >>>> >>>> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) >>>> at >>>> >>>> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) >>>> at >>>> scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108) >>>> at >>>> scala.collection.TraversableLike$class.map(TraversableLike.scala:244) >>>> at >>>> scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108) >>>> at >>>> >>>> org.apache.spark.sql.SQLContext$$anonfun$org$apache$spark$sql$SQLContext$$beansToRows$1.apply(SQLContext.scala:1358) >>>> at >>>> >>>> org.apache.spark.sql.SQLContext$$anonfun$org$apache$spark$sql$SQLContext$$beansToRows$1.apply(SQLContext.scala:1356) >>>> at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) >>>> at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) >>>> at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) >>>> at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) >>>> at scala.collection.Iterator$$anon$10.next(Iterator.scala:312) >>>> at scala.collection.Iterator$class.foreach(Iterator.scala:727) >>>> at >>>> scala.collection.AbstractIterator.foreach(Iterator.scala:1157) >>>> at >>>> scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48) >>>> at >>>> >>>> scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103) >>>> at >>>> scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47) >>>> at scala.collection.TraversableOnce$class.to >>>> (TraversableOnce.scala:273) >>>> at scala.collection.AbstractIterator.to(Iterator.scala:1157) >>>> at >>>> >>>> scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265) >>>> at >>>> scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157) >>>> at >>>> >>>> scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252) >>>> at >>>> scala.collection.AbstractIterator.toArray(Iterator.scala:1157) >>>> at >>>> >>>> org.apache.spark.sql.execution.SparkPlan$$anonfun$5.apply(SparkPlan.scala:212) >>>> at >>>> >>>> org.apache.spark.sql.execution.SparkPlan$$anonfun$5.apply(SparkPlan.scala:212) >>>> at >>>> >>>> org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858) >>>> at >>>> >>>> org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858) >>>> at >>>> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) >>>> at org.apache.spark.scheduler.Task.run(Task.scala:89) >>>> at >>>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213) >>>> at >>>> >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>>> at >>>> >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>>> at java.lang.Thread.run(Thread.java:745) >>>> >>>> >>>> >>>> -- >>>> View this message in context: >>>> http://apache-spark-user-list.1001560.n3.nabble.com/Scala-MatchError-in-Spark-SQL-tp26021.html >>>> Sent from the Apache Spark User List mailing list archive at Nabble.com. >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>>> For additional commands, e-mail: user-h...@spark.apache.org >>>> >>>> >>> >> >
