Hello there, I hope I get the whole thing correctly... but I think I did something quite similar that polls the yahoo api for data, which I push into a DStream: https://github.com/andypetrella/spark-bd/blob/master/src/main/scala/yahoo.scala#L41
Any comments or even concerns (if they are constructive ^^) are more than welcome. And, btw, I was thinking the same, this kind of use cases should be enabled easier in the API. Cheers andy On Wed, Sep 18, 2013 at 9:42 PM, Vadim Chekan <[email protected]> wrote: > Hi Paul, > > I had the same thoughts when started looking at spark. > This link might be helpful: > https://github.com/mesos/spark/blob/master/docs/streaming-custom-receivers.md > > Vadim. > > > > On Tue, Sep 17, 2013 at 6:50 PM, Paul Snively <[email protected]> wrote: > >> Hi everyone! >> >> In looking at the Spark Streaming API more closely, it occurs to me that >> it seems biased toward seeing Spark Streaming processes as consumers of >> existing services. I'm thinking particularly of socketStream and zeroMQ >> stream, which want a host/port and Publisher, respectively. But I think >> this overlooks a whole class of use cases, where I'd like to establish a >> server that someone can stream data to, and use Spark Streaming to process >> it. Has any work been done on this front? I suppose, worst case, I could >> write my own NetworkReceiver that listens on a socket or somesuch, but it'd >> be nice if I didn't have to. >> >> Thoughts? >> >> Thanks! >> Paul >> >> > > > -- > From RFC 2631: In ASN.1, EXPLICIT tagging is implicit unless IMPLICIT is > explicitly specified >
