Hi, See https://github.com/jaceklaskowski/spark-workshop/tree/master/solutions/spark-mf-format. There's a custom format that you can use to get started.
Basically, you need to develop the code behind "mysource" format and register it using --packages or --jars or similar when you spark-submit your Spark application. Pozdrawiam, Jacek Laskowski ---- https://medium.com/@jaceklaskowski/ Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark Follow me at https://twitter.com/jaceklaskowski On Sun, Jul 31, 2016 at 6:45 PM, Ayoub Benali <benali.ayoub.i...@gmail.com> wrote: > Looks like the way to go in spark 2.0 is to implement StreamSourceProvider > with DataSourceRegister. But now spark fails at loading the class when > doing: > > spark.readStream.format("mysource").load() > > I get : > > java.lang.ClassNotFoundException: Failed to find data source: mysource. > Please find packages at http://spark-packages.org > > Is there something I need to do in order to "load" the Stream source > provider ? > > Thanks, > Ayoub > > 2016-07-31 17:19 GMT+02:00 Jacek Laskowski <ja...@japila.pl>: >> >> On Sun, Jul 31, 2016 at 12:53 PM, Ayoub Benali >> <benali.ayoub.i...@gmail.com> wrote: >> >> > I started playing with the Structured Streaming API in spark 2.0 and I >> > am >> > looking for a way to create streaming Dataset/Dataframe from a rest HTTP >> > endpoint but I am bit stuck. >> >> What a great idea! Why did I myself not think about this?!?! >> >> > What would be the easiest way to hack around it ? Do I need to implement >> > the >> > Datasource API ? >> >> Yes and perhaps Hadoop API too, but not sure which one exactly since I >> haven't even thought about it (not even once). >> >> > Are there examples on how to create a DataSource from a REST endpoint ? >> >> Never heard of one. >> >> I'm hosting a Spark/Scala meetup this week so I'll definitely propose >> it as a topic. Thanks a lot! >> >> Pozdrawiam, >> Jacek Laskowski >> ---- >> https://medium.com/@jaceklaskowski/ >> Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark >> Follow me at https://twitter.com/jaceklaskowski > > --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
