Not sure I understand this problem, why I cannot reproduce it?
scala> spark.version
res22: String = 2.1.0
scala> case class Teamuser(teamid: String, userid: String, role: String)
defined class Teamuser
scala> val df = Seq(Teamuser("t1", "u1", "role1")).toDF
df: org.apache.spark.sql.DataFrame = [teamid: string, userid: string ... 1 more
field]
scala> df.show
+------+------+-----+
|teamid|userid| role|
+------+------+-----+
| t1| u1|role1|
+------+------+-----+
scala> df.createOrReplaceTempView("teamuser")
scala> val newDF = spark.sql("select teamid, userid, role from teamuser")
newDF: org.apache.spark.sql.DataFrame = [teamid: string, userid: string ... 1
more field]
scala> val userDS: Dataset[Teamuser] = newDF.as[Teamuser]
userDS: org.apache.spark.sql.Dataset[Teamuser] = [teamid: string, userid:
string ... 1 more field]
scala> userDS.show
+------+------+-----+
|teamid|userid| role|
+------+------+-----+
| t1| u1|role1|
+------+------+-----+
scala> userDS.printSchema
root
|-- teamid: string (nullable = true)
|-- userid: string (nullable = true)
|-- role: string (nullable = true)
Am I missing anything?
Yong
________________________________
From: shyla deshpande <[email protected]>
Sent: Thursday, March 23, 2017 3:49 PM
To: user
Subject: Re: Converting dataframe to dataset question
I realized, my case class was inside the object. It should be defined outside
the scope of the object. Thanks
On Wed, Mar 22, 2017 at 6:07 PM, shyla deshpande
<[email protected]<mailto:[email protected]>> wrote:
Why userDS is Dataset[Any], instead of Dataset[Teamuser]? Appreciate your
help. Thanks
val spark = SparkSession
.builder
.config("spark.cassandra.connection.host", cassandrahost)
.appName(getClass.getSimpleName)
.getOrCreate()
import spark.implicits._
val sqlContext = spark.sqlContext
import sqlContext.implicits._
case class Teamuser(teamid:String, userid:String, role:String)
spark
.read
.format("org.apache.spark.sql.cassandra")
.options(Map("keyspace" -> "test", "table" -> "teamuser"))
.load
.createOrReplaceTempView("teamuser")
val userDF = spark.sql("SELECT teamid, userid, role FROM teamuser")
userDF.show()
val userDS:Dataset[Teamuser] = userDF.as[Teamuser]
