Microbatching is certainly not a waste of time, you are making way too strong of an statement. In fact in certain cases one tuple at the time makes no sense, it all depends on the use cases. In fact if you understand the history of the project Storm you would know that microbatching was added later in Storm, Trident, and it is specifically for microbatching/windowing. In certain cases you are doing aggregation/windowing and throughput is the dominant design consideration and you don't care what each individual event/tuple does, e.g. of you push different event types to separate kafka topics and all you care is to do a count, what is the need for single event processing.
On Sun, Apr 17, 2016 at 12:43 PM, Corey Nolet <cjno...@gmail.com> wrote: > i have not been intrigued at all by the microbatching concept in Spark. I > am used to CEP in real streams processing environments like Infosphere > Streams & Storm where the granularity of processing is at the level of each > individual tuple and processing units (workers) can react immediately to > events being received and processed. The closest Spark streaming comes to > this concept is the notion of "state" that that can be updated via the > "updateStateBykey()" functions which are only able to be run in a > microbatch. Looking at the expected design changes to Spark Streaming in > Spark 2.0.0, it also does not look like tuple-at-a-time processing is on > the radar for Spark, though I have seen articles stating that more effort > is going to go into the Spark SQL layer in Spark streaming which may make > it more reminiscent of Esper. > > For these reasons, I have not even tried to implement CEP in Spark. I feel > it's a waste of time without immediate tuple-at-a-time processing. Without > this, they avoid the whole problem of "back pressure" (though keep in mind, > it is still very possible to overload the Spark streaming layer with stages > that will continue to pile up and never get worked off) but they lose the > granular control that you get in CEP environments by allowing the rules & > processors to react with the receipt of each tuple, right away. > > Awhile back, I did attempt to implement an InfoSphere Streams-like API [1] > on top of Apache Storm as an example of what such a design may look like. > It looks like Storm is going to be replaced in the not so distant future by > Twitter's new design called Heron. IIRC, Heron does not have an open source > implementation as of yet. > > [1] https://github.com/calrissian/flowmix > > On Sun, Apr 17, 2016 at 3:11 PM, Mich Talebzadeh < > mich.talebza...@gmail.com> wrote: > >> Hi Corey, >> >> Can you please point me to docs on using Spark for CEP? Do we have a set >> of CEP libraries somewhere. I am keen on getting hold of adaptor libraries >> for Spark something like below >> >> >> >> >> Thanks >> >> >> Dr Mich Talebzadeh >> >> >> >> LinkedIn * >> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >> >> >> >> http://talebzadehmich.wordpress.com >> >> >> >> On 17 April 2016 at 16:07, Corey Nolet <cjno...@gmail.com> wrote: >> >>> One thing I've noticed about Flink in my following of the project has >>> been that it has established, in a few cases, some novel ideas and >>> improvements over Spark. The problem with it, however, is that both the >>> development team and the community around it are very small and many of >>> those novel improvements have been rolled directly into Spark in subsequent >>> versions. I was considering changing over my architecture to Flink at one >>> point to get better, more real-time CEP streaming support, but in the end I >>> decided to stick with Spark and just watch Flink continue to pressure it >>> into improvement. >>> >>> On Sun, Apr 17, 2016 at 11:03 AM, Koert Kuipers <ko...@tresata.com> >>> wrote: >>> >>>> i never found much info that flink was actually designed to be fault >>>> tolerant. if fault tolerance is more bolt-on/add-on/afterthought then that >>>> doesn't bode well for large scale data processing. spark was designed with >>>> fault tolerance in mind from the beginning. >>>> >>>> On Sun, Apr 17, 2016 at 9:52 AM, Mich Talebzadeh < >>>> mich.talebza...@gmail.com> wrote: >>>> >>>>> Hi, >>>>> >>>>> I read the benchmark published by Yahoo. Obviously they already use >>>>> Storm and inevitably very familiar with that tool. To start with although >>>>> these benchmarks were somehow interesting IMO, it lend itself to an >>>>> assurance that the tool chosen for their platform is still the best >>>>> choice. >>>>> So inevitably the benchmarks and the tests were done to support >>>>> primary their approach. >>>>> >>>>> In general anything which is not done through TCP Council or similar >>>>> body is questionable.. >>>>> Their argument is that because Spark handles data streaming in micro >>>>> batches then inevitably it introduces this in-built latency as per design. >>>>> In contrast, both Storm and Flink do not (at the face value) have this >>>>> issue. >>>>> >>>>> In addition as we already know Spark has far more capabilities >>>>> compared to Flink (know nothing about Storm). So really it boils down to >>>>> the business SLA to choose which tool one wants to deploy for your use >>>>> case. IMO Spark micro batching approach is probably OK for 99% of use >>>>> cases. If we had in built libraries for CEP for Spark (I am searching for >>>>> it), I would not bother with Flink. >>>>> >>>>> HTH >>>>> >>>>> >>>>> Dr Mich Talebzadeh >>>>> >>>>> >>>>> >>>>> LinkedIn * >>>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>>>> >>>>> >>>>> >>>>> http://talebzadehmich.wordpress.com >>>>> >>>>> >>>>> >>>>> On 17 April 2016 at 12:47, Ovidiu-Cristian MARCU < >>>>> ovidiu-cristian.ma...@inria.fr> wrote: >>>>> >>>>>> You probably read this benchmark at Yahoo, any comments from Spark? >>>>>> >>>>>> https://yahooeng.tumblr.com/post/135321837876/benchmarking-streaming-computation-engines-at >>>>>> >>>>>> >>>>>> On 17 Apr 2016, at 12:41, andy petrella <andy.petre...@gmail.com> >>>>>> wrote: >>>>>> >>>>>> Just adding one thing to the mix: `that the latency for streaming >>>>>> data is eliminated` is insane :-D >>>>>> >>>>>> On Sun, Apr 17, 2016 at 12:19 PM Mich Talebzadeh < >>>>>> mich.talebza...@gmail.com> wrote: >>>>>> >>>>>>> It seems that Flink argues that the latency for streaming data is >>>>>>> eliminated whereas with Spark RDD there is this latency. >>>>>>> >>>>>>> I noticed that Flink does not support interactive shell much like >>>>>>> Spark shell where you can add jars to it to do kafka testing. The advice >>>>>>> was to add the streaming Kafka jar file to CLASSPATH but that does not >>>>>>> work. >>>>>>> >>>>>>> Most Flink documentation also rather sparce with the usual example >>>>>>> of word count which is not exactly what you want. >>>>>>> >>>>>>> Anyway I will have a look at it further. I have a Spark Scala >>>>>>> streaming Kafka program that works fine in Spark and I want to recode it >>>>>>> using Scala for Flink with Kafka but have difficulty importing and >>>>>>> testing >>>>>>> libraries. >>>>>>> >>>>>>> Cheers >>>>>>> >>>>>>> Dr Mich Talebzadeh >>>>>>> >>>>>>> >>>>>>> LinkedIn * >>>>>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>>>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>>>>>> >>>>>>> >>>>>>> http://talebzadehmich.wordpress.com >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 17 April 2016 at 02:41, Ascot Moss <ascot.m...@gmail.com> wrote: >>>>>>> >>>>>>>> I compared both last month, seems to me that Flink's MLLib is not >>>>>>>> yet ready. >>>>>>>> >>>>>>>> On Sun, Apr 17, 2016 at 12:23 AM, Mich Talebzadeh < >>>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>>> >>>>>>>>> Thanks Ted. I was wondering if someone is using both :) >>>>>>>>> >>>>>>>>> Dr Mich Talebzadeh >>>>>>>>> >>>>>>>>> >>>>>>>>> LinkedIn * >>>>>>>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>>>>>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>>>>>>>> >>>>>>>>> >>>>>>>>> http://talebzadehmich.wordpress.com >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On 16 April 2016 at 17:08, Ted Yu <yuzhih...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Looks like this question is more relevant on flink mailing list >>>>>>>>>> :-) >>>>>>>>>> >>>>>>>>>> On Sat, Apr 16, 2016 at 8:52 AM, Mich Talebzadeh < >>>>>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> Has anyone used Apache Flink instead of Spark by any chance >>>>>>>>>>> >>>>>>>>>>> I am interested in its set of libraries for Complex Event >>>>>>>>>>> Processing. >>>>>>>>>>> >>>>>>>>>>> Frankly I don't know if it offers far more than Spark offers. >>>>>>>>>>> >>>>>>>>>>> Thanks >>>>>>>>>>> >>>>>>>>>>> Dr Mich Talebzadeh >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> LinkedIn * >>>>>>>>>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>>>>>>>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> http://talebzadehmich.wordpress.com >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> -- >>>>>> andy >>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> >
