Re: Converting dataframe to dataset question

2017-03-23 Thread shyla deshpande
; >>> scala> val userDS: Dataset[Teamuser] = newDF.as[Teamuser] >>> userDS: org.apache.spark.sql.Dataset[Teamuser] = [teamid: string, userid: >>> string ... 1 more field] >>> >>> scala> userDS.show >>> +--+--+-+ >>> |t

Re: Converting dataframe to dataset question

2017-03-23 Thread Ryan
. 1 more field] >> >> scala> userDS.show >> +--+--+-+ >> |teamid|userid| role| >> +--+------+-----+ >> |t1|u1|role1| >> +--+--+-+ >> >> >> scala> userDS.printSchema >> root >> |-- teami

Re: Converting dataframe to dataset question

2017-03-23 Thread shyla deshpande
userDS.show > +--+--+-+ > |teamid|userid| role| > +--+--+-+ > |t1|u1|role1| > +--+--+-+ > > > scala> userDS.printSchema > root > |-- teamid: string (nullable = true) > |-- userid: string (nullable = true) > |-- ro

Re: Converting dataframe to dataset question

2017-03-23 Thread shyla deshpande
now I get a run time error... error: Unable to find encoder for type stored in a Dataset. Primitive types (Int, String, etc) and Product types (case classes) are supported by importing spark.implicits._ Support for serializing other types will be added in future releases. [ERROR] val

Re: Converting dataframe to dataset question

2017-03-23 Thread Yong Zhang
true) Am I missing anything? Yong From: shyla deshpande <deshpandesh...@gmail.com> Sent: Thursday, March 23, 2017 3:49 PM To: user Subject: Re: Converting dataframe to dataset question I realized, my case class was inside the object. It should be defin

Re: Converting dataframe to dataset question

2017-03-23 Thread shyla deshpande
I realized, my case class was inside the object. It should be defined outside the scope of the object. Thanks On Wed, Mar 22, 2017 at 6:07 PM, shyla deshpande wrote: > Why userDS is Dataset[Any], instead of Dataset[Teamuser]? Appreciate your > help. Thanks > >

Converting dataframe to dataset question

2017-03-22 Thread shyla deshpande
Why userDS is Dataset[Any], instead of Dataset[Teamuser]? Appreciate your help. Thanks val spark = SparkSession .builder .config("spark.cassandra.connection.host", cassandrahost) .appName(getClass.getSimpleName) .getOrCreate() import spark.implicits._ val