Hi Esa, I am trying to use Spark streaming for CEP and is looking promising.
*General (from my **blog * <https://talebzadehmich.wordpress.com/category/complex-event-processing/>) In a nutshell CEP involves the continuous processing and analysis of high-volume, high-speed data streams from inside and outside of an enterprise to detect business-critical issues as they happen in real time. Contrast this to the traditional processes involving database systems, which provide delayed analysis. An example of CEP would be real-time financial market data analysis and decision process allowing traders or anyone else to make a decision on the spot based on real time data. Prime example is a Forex System where on the basis of certain indicators (say moving averages for the past 14 periods) you make a decision to buy or sell. CEP software offers two major components: a high-level language for programmers to easily describe how to process the streams/messages, and an infrastructure engine for processing and analyzing high-volume data Although CEP software performs different functions, the component structure is somehow analogous to database software, where there is a language (say SQL) and an engine (the database server). The objectives of CEP is to get the product and save on development cycle traditionally done by in-house developers. Your points: Is CEP only for (real time) stream data and not for "history" data? Generally real time as you make decisions on the spot say buy or sell in Forex or Algorithmic trading. From business point of view that is the most valuable. You can of course store data of interest in a DW like Hive for historical analysis (trend analytics). 2) Is it possible to search "backward" (upstream) by CEP with given time window? If a start time of the time window is earlier than the current stream time. I haver not tried this but I believe it is feasible using something like Kafka broker which retains events for a period of time. However, I don't see much of business value for it. 3) Do you know any good tools or softwares for "CEP's" using for log data ? What is the definition of log data here? Are you referring to producers here? 4) Do you know any good (scientific) papers i should read about CEP ? There are many references in web as Mario pointed out. However, as a student you need to grasp the need for Complex Event Processing which is Event Driven Architecture. A good classic book on this is "The Power of Events" by David Luckham, ISBN 0-201-72789-7. In general the architecture I am aiming is Kafka,Zookeeper and Spark Streaming. HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com On 27 April 2016 at 11:02, Mario Ds Briggs <mario.bri...@in.ibm.com> wrote: > Wikipedia defines the goal of CEP as 'respond to them (events) as quickly > as possible' . So i think there is an direct link to 'streaming', when we > say CEP. > > However pattern matching at its core applies even on historical data. > > > thanks > Mario > > > ----- Message from Esa Heikkinen <esa.heikki...@student.tut.fi> on Wed, > 27 Apr 2016 12:40:52 +0300 ----- > > *To:* > user@spark.apache.org > > *Subject:* > Re: Spark support for Complex Event Processing (CEP)Hi > > I have followed with interest the discussion about CEP and Spark. It is > quite close to my research, which is a complex analyzing for log files > and "history" data (not actually for real time streams). > > I have few questions: > > 1) Is CEP only for (real time) stream data and not for "history" data? > > 2) Is it possible to search "backward" (upstream) by CEP with given time > window? If a start time of the time window is earlier than the current > stream time. > > 3) Do you know any good tools or softwares for "CEP's" using for log data ? > > 4) Do you know any good (scientific) papers i should read about CEP ? > > > Regards > PhD student at Tampere University of Technology, Finland, www.tut.fi > Esa Heikkinen > > > > >
