You have to add a file in resource too (example <https://github.com/apache/spark/blob/master/sql/core/src/main/resources/META-INF/services/org.apache.spark.sql.sources.DataSourceRegister>). Either that or give a full class name.
On Sun, Jul 31, 2016 at 9:45 AM, Ayoub Benali <benali.ayoub.i...@gmail.com> wrote: > Looks like the way to go in spark 2.0 is to implement StreamSourceProvider > <https://github.com/apache/spark/blob/9e2c763dbb5ac6fc5d2eb0759402504d4b9073a4/sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala#L117> > with DataSourceRegister > <https://github.com/apache/spark/blob/9e2c763dbb5ac6fc5d2eb0759402504d4b9073a4/sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala#L40>. > But now spark fails at loading the class when doing: > > spark.readStream.format("mysource").load() > > I get : > > java.lang.ClassNotFoundException: Failed to find data source: mysource. > Please find packages at http://spark-packages.org > > Is there something I need to do in order to "load" the Stream source > provider ? > > Thanks, > Ayoub > > 2016-07-31 17:19 GMT+02:00 Jacek Laskowski <ja...@japila.pl>: > >> On Sun, Jul 31, 2016 at 12:53 PM, Ayoub Benali >> <benali.ayoub.i...@gmail.com> wrote: >> >> > I started playing with the Structured Streaming API in spark 2.0 and I >> am >> > looking for a way to create streaming Dataset/Dataframe from a rest HTTP >> > endpoint but I am bit stuck. >> >> What a great idea! Why did I myself not think about this?!?! >> >> > What would be the easiest way to hack around it ? Do I need to >> implement the >> > Datasource API ? >> >> Yes and perhaps Hadoop API too, but not sure which one exactly since I >> haven't even thought about it (not even once). >> >> > Are there examples on how to create a DataSource from a REST endpoint ? >> >> Never heard of one. >> >> I'm hosting a Spark/Scala meetup this week so I'll definitely propose >> it as a topic. Thanks a lot! >> >> Pozdrawiam, >> Jacek Laskowski >> ---- >> https://medium.com/@jaceklaskowski/ >> Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark >> Follow me at https://twitter.com/jaceklaskowski >> > >
