Hi Tamas, I meant not changing the receivers, but starting/stopping the Streaming jobs. So you would have a 'small' Streaming job for a subset of streams that you'd configure->start->stop on demand. I haven't tried myself yet, but I think it should also be possible to create a Streaming Job from the Spark Job Server ( https://github.com/spark-jobserver/spark-jobserver). Then you would have a REST interface that even gives you the possibility of passing a configuration.
-kr, Gerard. On Wed, Jan 21, 2015 at 11:54 AM, Tamas Jambor <jambo...@gmail.com> wrote: > we were thinking along the same line, that is to fix the number of streams > and change the input and output channels dynamically. > > But could not make it work (seems that the receiver is not allowing any > change in the config after it started). > > thanks, > > On Wed, Jan 21, 2015 at 10:49 AM, Gerard Maas <gerard.m...@gmail.com> > wrote: > >> One possible workaround could be to orchestrate launch/stopping of >> Streaming jobs on demand as long as the number of jobs/streams stay >> within the boundaries of the resources (cores) you've available. >> e.g. if you're using Mesos, Marathon offers a REST interface to manage >> job lifecycle. You will still need to solve the dynamic configuration >> through some alternative channel. >> >> On Wed, Jan 21, 2015 at 11:30 AM, Tamas Jambor <jambo...@gmail.com> >> wrote: >> >>> thanks for the replies. >>> >>> is this something we can get around? Tried to hack into the code without >>> much success. >>> >>> On Wed, Jan 21, 2015 at 3:15 AM, Shao, Saisai <saisai.s...@intel.com> >>> wrote: >>> >>>> Hi, >>>> >>>> I don't think current Spark Streaming support this feature, all the >>>> DStream lineage is fixed after the context is started. >>>> >>>> Also stopping a stream is not supported, instead currently we need to >>>> stop the whole streaming context to meet what you want. >>>> >>>> Thanks >>>> Saisai >>>> >>>> -----Original Message----- >>>> From: jamborta [mailto:jambo...@gmail.com] >>>> Sent: Wednesday, January 21, 2015 3:09 AM >>>> To: user@spark.apache.org >>>> Subject: dynamically change receiver for a spark stream >>>> >>>> Hi all, >>>> >>>> we have been trying to setup a stream using a custom receiver that >>>> would pick up data from sql databases. we'd like to keep that stream >>>> context running and dynamically change the streams on demand, adding and >>>> removing streams based on demand. alternativel, if a stream is fixed, is it >>>> possible to stop a stream, change to config and start again? >>>> >>>> thanks, >>>> >>>> >>>> >>>> -- >>>> View this message in context: >>>> http://apache-spark-user-list.1001560.n3.nabble.com/dynamically-change-receiver-for-a-spark-stream-tp21268.html >>>> Sent from the Apache Spark User List mailing list archive at Nabble.com. >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For >>>> additional commands, e-mail: user-h...@spark.apache.org >>>> >>>> >>> >> >
