Thanks Vincent for the feedback. I appreciate. On Tue, Nov 29, 2016 at 1:34 AM, vincent gromakowski < vincent.gromakow...@gmail.com> wrote:
> You can still achieve it by implementing an actor in each partition but I > am not sure it's a good design regarding scalability because your > distributed actors would send a message for each event to your single app > actor, it would be a huge load.... > If you want to experiment this and because actor is thread safe you can > use the following pattern which allows to reuse actors between micro > batches in each partitions > http://allegro.tech/2015/08/spark-kafka-integration.html > > > 2016-11-29 2:18 GMT+01:00 shyla deshpande <deshpandesh...@gmail.com>: > >> Hello All, >> >> I just want to make sure this is a right use case for Kafka --> Spark >> Streaming >> >> Few words about my use case : >> >> When the user watches a video, I get the position events from the user >> that indicates how much they have completed viewing and at a certain point, >> I mark that Video as complete and that triggers a lot of other events. I >> need a way to notify the app about the creation of the completion event. >> >> Appreciate any suggestions. >> >> Thanks >> >> >> On Mon, Nov 28, 2016 at 2:35 PM, shyla deshpande < >> deshpandesh...@gmail.com> wrote: >> >>> In this case, persisting to Cassandra is for future analytics and >>> Visualization. >>> >>> I want to notify that the app of the event, so it makes the app >>> interactive. >>> >>> Thanks >>> >>> On Mon, Nov 28, 2016 at 2:24 PM, vincent gromakowski < >>> vincent.gromakow...@gmail.com> wrote: >>> >>>> Sorry I don't understand... >>>> Is it a cassandra acknowledge to actors that you want ? Why do you want >>>> to ack after writing to cassandra ? Your pipeline kafka=>spark=>cassandra >>>> is supposed to be exactly once, so you don't need to wait for cassandra >>>> ack, you can just write to kafka from actors and then notify the user ? >>>> >>>> 2016-11-28 23:15 GMT+01:00 shyla deshpande <deshpandesh...@gmail.com>: >>>> >>>>> Thanks Vincent for the input. Not sure I understand your suggestion. >>>>> Please clarify. >>>>> >>>>> Few words about my use case : >>>>> When the user watches a video, I get the position events from the user >>>>> that indicates how much they have completed viewing and at a certain >>>>> point, >>>>> I mark that Video as complete and persist that info to cassandra. >>>>> >>>>> How do I notify the user that it was marked complete? >>>>> >>>>> Are you suggesting I write the completed events to kafka(different >>>>> topic) and the akka consumer could read from this? There could be many >>>>> completed events from different users in this topic. So the akka consumer >>>>> should pretty much do what a spark streaming does to process this without >>>>> the knowledge of the kafka offset. >>>>> >>>>> So not sure what you mean by kafka offsets will do the job, how will >>>>> the akka consumer know the kafka offset? >>>>> >>>>> On Mon, Nov 28, 2016 at 12:52 PM, vincent gromakowski < >>>>> vincent.gromakow...@gmail.com> wrote: >>>>> >>>>>> You don't need actors to do kafka=>spark processing=>kafka >>>>>> Why do you need to notify the akka producer ? If you need to get back >>>>>> the processed message in your producer, then implement an akka consummer >>>>>> in >>>>>> your akka app and kafka offsets will do the job >>>>>> >>>>>> 2016-11-28 21:46 GMT+01:00 shyla deshpande <deshpandesh...@gmail.com> >>>>>> : >>>>>> >>>>>>> Thanks Daniel for the response. >>>>>>> >>>>>>> I am planning to use Spark streaming to do Event Processing. I will >>>>>>> have akka actors sending messages to kafka. I process them using Spark >>>>>>> streaming and as a result a new events will be generated. How do I >>>>>>> notify >>>>>>> the akka actor(Message producer) that a new event has been generated? >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Mon, Nov 28, 2016 at 9:51 AM, Daniel van der Ende < >>>>>>> daniel.vandere...@gmail.com> wrote: >>>>>>> >>>>>>>> Well, I would say it depends on what you're trying to >>>>>>>> achieve. Right now I don't know why you are considering using Akka. >>>>>>>> Could >>>>>>>> you please explain your use case a bit? >>>>>>>> >>>>>>>> In general, there is no single correct answer to your current >>>>>>>> question as it's quite broad. >>>>>>>> >>>>>>>> Daniel >>>>>>>> >>>>>>>> On Mon, Nov 28, 2016 at 9:11 AM, shyla deshpande < >>>>>>>> deshpandesh...@gmail.com> wrote: >>>>>>>> >>>>>>>>> My data pipeline is Kafka --> Spark Streaming --> Cassandra. >>>>>>>>> >>>>>>>>> Can someone please explain me when would I need to wrap akka >>>>>>>>> around the spark streaming app. My knowledge of akka and the actor >>>>>>>>> system >>>>>>>>> is poor. Please help! >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Daniel >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
