couple of things. There is no such thing as Continuous Data Streaming as there is no such thing as Continuous Availability.
There is such thing as Discrete Data Streaming and High Availability but they reduce the finite unavailability to minimum. In terms of business needs a 5 SIGMA is good enough and acceptable. Even the candles set to a predefined time interval say 2, 4, 15 seconds overlap. No FX savvy trader makes a sell or buy decision on the basis of 2 seconds candlestick The calculation itself in measurements is subject to finite error as defined by their Confidence Level (CL) using Standard Deviation function. OK so far I have never noticed a tool that requires that details of granularity. Those stuff from Flink etc is in practical term is of little value and does not make commercial sense. Now with regard to your needs, Spark micro batching is perfectly adequate. HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com On 27 April 2016 at 22:10, Esa Heikkinen <esa.heikki...@student.tut.fi> wrote: > > Hi > > Thanks for the answer. > > I have developed a log file analyzer for RTPIS (Real Time Passenger > Information System) system, where buses drive lines and the system try to > estimate the arrival times to the bus stops. There are many different log > files (and events) and analyzing situation can be very complex. Also > spatial data can be included to the log data. > > The analyzer also has a query (or analyzing) language, which describes a > expected behavior. This can be a requirement of system. Analyzer can be > think to be also a test oracle. > > I have published a paper (SPLST'15 conference) about my analyzer and its > language. The paper is maybe too technical, but it is found: > http://ceur-ws.org/Vol-1525/paper-19.pdf > > I do not know yet where it belongs. I think it can be some "CEP with > delays". Or do you know better ? > My analyzer can also do little bit more complex and time-consuming > analyzings? There are no a need for real time. > > And it is possible to do "CEP with delays" reasonably some existing > analyzer (for example Spark) ? > > Regards > PhD student at Tampere University of Technology, Finland, www.tut.fi > Esa Heikkinen > > 27.4.2016, 15:51, Michael Segel kirjoitti: > > Spark and CEP? It depends… > > Ok, I know that’s not the answer you want to hear, but its a bit more > complicated… > > If you consider Spark Streaming, you have some issues. > Spark Streaming isn’t a Real Time solution because it is a micro batch > solution. The smallest Window is 500ms. This means that if your compute > time is >> 500ms and/or your event flow is >> 500ms this could work. > (e.g. 'real time' image processing on a system that is capturing 60FPS > because the processing time is >> 500ms. ) > > So Spark Streaming wouldn’t be the best solution…. > > However, you can combine spark with other technologies like Storm, Akka, > etc .. where you have continuous streaming. > So you could instantiate a spark context per worker in storm… > > I think if there are no class collisions between Akka and Spark, you could > use Akka, which may have a better potential for communication between > workers. > So here you can handle CEP events. > > HTH > > On Apr 27, 2016, at 7:03 AM, Mich Talebzadeh <mich.talebza...@gmail.com> > wrote: > > please see my other reply > > Dr Mich Talebzadeh > > > LinkedIn * > <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* > > > http://talebzadehmich.wordpress.com > > > > On 27 April 2016 at 10:40, Esa Heikkinen <esa.heikki...@student.tut.fi> > wrote: > >> Hi >> >> I have followed with interest the discussion about CEP and Spark. It is >> quite close to my research, which is a complex analyzing for log files and >> "history" data (not actually for real time streams). >> >> I have few questions: >> >> 1) Is CEP only for (real time) stream data and not for "history" data? >> >> 2) Is it possible to search "backward" (upstream) by CEP with given time >> window? If a start time of the time window is earlier than the current >> stream time. >> >> 3) Do you know any good tools or softwares for "CEP's" using for log data >> ? >> >> 4) Do you know any good (scientific) papers i should read about CEP ? >> >> >> Regards >> PhD student at Tampere University of Technology, Finland, www.tut.fi >> Esa Heikkinen >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> >> > > The opinions expressed here are mine, while they may reflect a cognitive > thought, that is purely accidental. > Use at your own risk. > Michael Segel > michael_segel (AT) hotmail.com > > > > > > >
