Looks like the way to go in spark 2.0 is to implement StreamSourceProvider <https://github.com/apache/spark/blob/9e2c763dbb5ac6fc5d2eb0759402504d4b9073a4/sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala#L117> with DataSourceRegister <https://github.com/apache/spark/blob/9e2c763dbb5ac6fc5d2eb0759402504d4b9073a4/sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala#L40>. But now spark fails at loading the class when doing:
spark.readStream.format("mysource").load() I get : java.lang.ClassNotFoundException: Failed to find data source: mysource. Please find packages at http://spark-packages.org Is there something I need to do in order to "load" the Stream source provider ? Thanks, Ayoub 2016-07-31 17:19 GMT+02:00 Jacek Laskowski <ja...@japila.pl>: > On Sun, Jul 31, 2016 at 12:53 PM, Ayoub Benali > <benali.ayoub.i...@gmail.com> wrote: > > > I started playing with the Structured Streaming API in spark 2.0 and I am > > looking for a way to create streaming Dataset/Dataframe from a rest HTTP > > endpoint but I am bit stuck. > > What a great idea! Why did I myself not think about this?!?! > > > What would be the easiest way to hack around it ? Do I need to implement > the > > Datasource API ? > > Yes and perhaps Hadoop API too, but not sure which one exactly since I > haven't even thought about it (not even once). > > > Are there examples on how to create a DataSource from a REST endpoint ? > > Never heard of one. > > I'm hosting a Spark/Scala meetup this week so I'll definitely propose > it as a topic. Thanks a lot! > > Pozdrawiam, > Jacek Laskowski > ---- > https://medium.com/@jaceklaskowski/ > Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark > Follow me at https://twitter.com/jaceklaskowski >