Try this: *import **spark*.implicits._
df.toDF() On Wed, Sep 5, 2018 at 2:31 PM Mich Talebzadeh <mich.talebza...@gmail.com> wrote: > With the following > > case class columns(KEY: String, TICKER: String, TIMEISSUED: String, PRICE: > Float) > > var key = line._2.split(',').view(0).toString > var ticker = line._2.split(',').view(1).toString > var timeissued = line._2.split(',').view(2).toString > var price = line._2.split(',').view(3).toFloat > > var df = Seq(columns(key, ticker, timeissued, price)) > println(df) > > I get > > > List(columns(ac11a78d-82df-4b37-bf58-7e3388aa64cd,MKS,2018-09-05T10:10:15,676.5)) > > So just need to convert that list to DF > > Dr Mich Talebzadeh > > > > LinkedIn * > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* > > > > http://talebzadehmich.wordpress.com > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > > > On Wed, 5 Sep 2018 at 09:49, Mich Talebzadeh <mich.talebza...@gmail.com> > wrote: > >> Thanks! >> >> The spark is version 2.3.0 >> >> Dr Mich Talebzadeh >> >> >> >> LinkedIn * >> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >> >> >> >> http://talebzadehmich.wordpress.com >> >> >> *Disclaimer:* Use it at your own risk. Any and all responsibility for >> any loss, damage or destruction of data or any other property which may >> arise from relying on this email's technical content is explicitly >> disclaimed. The author will in no case be liable for any monetary damages >> arising from such loss, damage or destruction. >> >> >> >> >> On Wed, 5 Sep 2018 at 09:41, Jungtaek Lim <kabh...@gmail.com> wrote: >> >>> You may also find below link useful (though it looks far old), since >>> case class is the thing which Encoder is available, so there may be another >>> reason which prevent implicit conversion. >>> >>> >>> https://community.cloudera.com/t5/Advanced-Analytics-Apache-Spark/Spark-Scala-Error-value-toDF-is-not-a-member-of-org-apache/m-p/29994/highlight/true#M973 >>> >>> And which Spark version do you use? >>> >>> >>> 2018년 9월 5일 (수) 오후 5:32, Jungtaek Lim <kabh...@gmail.com>님이 작성: >>> >>>> Sorry I guess I pasted another method. the code is... >>>> >>>> implicit def localSeqToDatasetHolder[T : Encoder](s: Seq[T]): >>>> DatasetHolder[T] = { >>>> DatasetHolder(_sqlContext.createDataset(s)) >>>> } >>>> >>>> >>>> 2018년 9월 5일 (수) 오후 5:30, Jungtaek Lim <kabh...@gmail.com>님이 작성: >>>> >>>>> I guess you need to have encoder for the type of result for columns(). >>>>> >>>>> >>>>> https://github.com/apache/spark/blob/2119e518d31331e65415e0f817a6f28ff18d2b42/sql/core/src/main/scala/org/apache/spark/sql/SQLImplicits.scala#L227-L229 >>>>> >>>>> implicit def rddToDatasetHolder[T : Encoder](rdd: RDD[T]): >>>>> DatasetHolder[T] = { >>>>> DatasetHolder(_sqlContext.createDataset(rdd)) >>>>> } >>>>> >>>>> You can see lots of Encoder implementations in the scala code. If your >>>>> type doesn't match anything it may not work and you need to provide custom >>>>> Encoder. >>>>> >>>>> -Jungtaek Lim (HeartSaVioR) >>>>> >>>>> 2018년 9월 5일 (수) 오후 5:24, Mich Talebzadeh <mich.talebza...@gmail.com>님이 >>>>> 작성: >>>>> >>>>>> Thanks >>>>>> >>>>>> I already do that as below >>>>>> >>>>>> val sqlContext= new org.apache.spark.sql.SQLContext(sparkContext) >>>>>> import sqlContext.implicits._ >>>>>> >>>>>> but still getting the error! >>>>>> >>>>>> Dr Mich Talebzadeh >>>>>> >>>>>> >>>>>> >>>>>> LinkedIn * >>>>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>>>>> >>>>>> >>>>>> >>>>>> http://talebzadehmich.wordpress.com >>>>>> >>>>>> >>>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility >>>>>> for any loss, damage or destruction of data or any other property which >>>>>> may >>>>>> arise from relying on this email's technical content is explicitly >>>>>> disclaimed. The author will in no case be liable for any monetary damages >>>>>> arising from such loss, damage or destruction. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Wed, 5 Sep 2018 at 09:17, Jungtaek Lim <kabh...@gmail.com> wrote: >>>>>> >>>>>>> You may need to import implicits from your spark session like below: >>>>>>> (Below code is borrowed from >>>>>>> https://spark.apache.org/docs/latest/sql-programming-guide.html) >>>>>>> >>>>>>> import org.apache.spark.sql.SparkSession >>>>>>> val spark = SparkSession >>>>>>> .builder() >>>>>>> .appName("Spark SQL basic example") >>>>>>> .config("spark.some.config.option", "some-value") >>>>>>> .getOrCreate() >>>>>>> // For implicit conversions like converting RDDs to DataFramesimport >>>>>>> spark.implicits._ >>>>>>> >>>>>>> >>>>>>> 2018년 9월 5일 (수) 오후 5:11, Mich Talebzadeh <mich.talebza...@gmail.com>님이 >>>>>>> 작성: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I have spark streaming that send data and I need to put that data >>>>>>>> into MongoDB for test purposes. The easiest way is to create a DF from >>>>>>>> the >>>>>>>> individual list of columns as below >>>>>>>> >>>>>>>> I loop over individual rows in RDD and perform the following >>>>>>>> >>>>>>>> case class columns(KEY: String, TICKER: String, TIMEISSUED: >>>>>>>> String, PRICE: Float) >>>>>>>> >>>>>>>> for(line <- pricesRDD.collect.toArray) >>>>>>>> { >>>>>>>> var key = line._2.split(',').view(0).toString >>>>>>>> var ticker = line._2.split(',').view(1).toString >>>>>>>> var timeissued = line._2.split(',').view(2).toString >>>>>>>> var price = line._2.split(',').view(3).toFloat >>>>>>>> val priceToString = line._2.split(',').view(3) >>>>>>>> if (price > 90.0) >>>>>>>> { >>>>>>>> println ("price > 90.0, saving to MongoDB collection!") >>>>>>>> // Save prices to mongoDB collection >>>>>>>> * var df = Seq(columns(key, ticker, timeissued, >>>>>>>> price)).toDF* >>>>>>>> >>>>>>>> but it fails with message >>>>>>>> >>>>>>>> value toDF is not a member of Seq[columns]. >>>>>>>> >>>>>>>> What would be the easiest way of resolving this please? >>>>>>>> >>>>>>>> thanks >>>>>>>> >>>>>>>> Dr Mich Talebzadeh >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> LinkedIn * >>>>>>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>>>>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> http://talebzadehmich.wordpress.com >>>>>>>> >>>>>>>> >>>>>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility >>>>>>>> for any loss, damage or destruction of data or any other property >>>>>>>> which may >>>>>>>> arise from relying on this email's technical content is explicitly >>>>>>>> disclaimed. The author will in no case be liable for any monetary >>>>>>>> damages >>>>>>>> arising from such loss, damage or destruction. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> -- Thanks Deepak www.bigdatabig.com www.keosha.net