[Dev] State persistence of the operators extending AbstractStreamProcessor in partitioned queries
Hi, I have come across the issue of the state of the AbstractStreamProcessor in partitioned queries not getting persisted in Siddhi. The issue has been explained with more details in [1]. In the solution which I have suggested [2] there will be an API change to the StreamRuntime interface [3]. This API change will basically add query name as a parameter to the clone() method of StreamRuntime. Propagating the query name via clone() as a parameter will allow us to properly add the AbstractStreamProcessor instances (which are to be persisted) to the snapshotableMap HashMap located in the SnapshotService [4] class. Furthermore, in the AbstractStreamProcessor class we will generate and assign a new elementID when cloning an instance out of it. We also need to access SiddhiAppContext object in the SingleStreamRuntime class in order to add the newly cloned AbstractStreamProcessor instances to the snapshotableMap. Hence we pass a SiddhiAppContext object as a parameter in the SingleStreamRuntime class's constructor. @Suho, We need to discuss and decide whether there will be significant impact of doing such API level change for the StreamRuntime interface. An alternative would be (as Tishan suggested in [2]), rather than modifying the existing clone() method parameters, we could add a second clone() method with both query name and partition key as the parameters. WDYT? [1] https://github.com/wso2/siddhi/issues/774 [2] https://github.com/wso2/siddhi/pull/775 [3] https://github.com/wso2/siddhi/blob/master/modules/siddhi-core/src/main/java/org/wso2/siddhi/core/query/input/stream/StreamRuntime.java [4] https://github.com/wso2/siddhi/blob/master/modules/siddhi-core/src/main/java/org/wso2/siddhi/core/util/snapshot/SnapshotService.java -- Thanks, Miyuru Dayarathna Senior Technical Lead Mobile: +94713527783 Blog: http://miyurublog.blogspot.com ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [GSOC 2017][CEP]Video Scene Detection with WSO2 CEP
Hi Hasitha, Thanks for your interest on video scene detection project. The objective of this project is to implement a Java-based video scene/shot detection system. The project will also investigate on the effective usage of Tensorflow for video scene/shot detection. Finally, the project will do an evaluation of the accuracy of the video scene detection by comparing results with few standard video benchmark datasets such as TRECVID. Please go through the following link to get you familiarized with the Tensorflow technology which we will use in this project [1]. [1] https://www.tensorflow.org/get_started/get_started Thanks, Miyuru Dayarathna, Ph.D. Senior Technical Lead, WSO2 Inc. On Tue, Mar 7, 2017 at 11:28 AM, Hasitha Jayasundara < hasithajayasunda...@gmail.com> wrote: > Dear Sir, > > I am a final year undergraduate of Electronic and Telecommunication > Engineering, University of Moratuwa. My experience with Java 8,C,algorithms > and WSo2 CEP[1] motivated me to select this project as my GSoC project for > 2017.I was an intern at WSO2 for 6 months and successfully completed Image > processing module(4th sem) in university.Please provide me some guide > lines and additional materials for study and get a clear understanding > about the mentioned project. > > Thank you > > [1] http://wso2.com/products/complex-event-processor/ > -- > *D.M.Hasitha Nadishan Jayasundara* > *Department of Electronic and Telecommunication Engineering* > University of Moratuwa > *mobile:* *+94711959266 <+94%2071%20195%209266>* > *blogspot*: learnjavawithisla.blogspot.com/ *email:* hasithajayasundar > a...@gmail.com > <https://www.facebook.com/hasitha.nadishan.3> > <https://twitter.com/HNadishan> > <https://lk.linkedin.com/in/hasithajayasundara> > -- Thanks, Miyuru Dayarathna Senior Technical Lead Mobile: +94713527783 Blog: http://miyurublog.blogspot.com ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] GSoC 2017
Hi Irene, Thanks for your interest on video scene detection project. The objective of this project is to implement a Java/OpenCV-based video scene/shot detection system. The project will also investigate on the effective usage of Tensorflow for video scene/shot detection. Finally, the project will do an evaluation of the accuracy of the video scene detection by comparing results with few standard video benchmark datasets such as TRECVID. Please go through the following two links to get you familiarized with the areas which we will be covering in this project [1][2]. [1] https://www.tensorflow.org/get_started/get_started [2] http://docs.opencv.org/3.2.0/d9/df8/tutorial_root.html Thanks, Miyuru Dayarathna, Ph.D. Senior Technical Lead, WSO2 Inc. On Tue, Feb 14, 2017 at 10:51 AM, Tishan Dahanayakage <tis...@wso2.com> wrote: > Adding again > > On Tue, Feb 14, 2017 at 10:38 AM, Tishan Dahanayakage <tis...@wso2.com> > wrote: > >> Hi Irene, >> >> Thanks for your interest in WSO2 and this project. I am adding possible >> project mentors here so that you can get more information. >> >> Thanks >> /Tishan >> >> On Tue, Feb 7, 2017 at 10:34 PM, irene tenison <irene.teni...@gmail.com> >> wrote: >> >>> I would like to contribute to WSO2 on video scene detection project for >>> GSoC 2017.Please do help me and guide me with the project. About myself .. >>> I'm Irene from India and I'm quite interested in video editing and colour >>> grading works and so I think this project would be most appropriately >>> handled by me.I would be so thankful if you can provide more information >>> about what you are trying to implement. >>> >>> ___ >>> Dev mailing list >>> Dev@wso2.org >>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>> >>> >> >> >> -- >> Tishan Dahanayakage >> Senior Software Engineer >> WSO2, Inc. >> Mobile:+94 716481328 <+94%2071%20648%201328> >> >> Disclaimer: This communication may contain privileged or other >> confidential information and is intended exclusively for the addressee/s. >> If you are not the intended recipient/s, or believe that you may have >> received this communication in error, please reply to the sender indicating >> that fact and delete the copy you received and in addition, you should not >> print, copy, re-transmit, disseminate, or otherwise use the information >> contained in this communication. Internet communications cannot be >> guaranteed to be timely, secure, error or virus-free. The sender does not >> accept liability for any errors or omissions. >> > > > > -- > Tishan Dahanayakage > Senior Software Engineer > WSO2, Inc. > Mobile:+94 716481328 <+94%2071%20648%201328> > > Disclaimer: This communication may contain privileged or other > confidential information and is intended exclusively for the addressee/s. > If you are not the intended recipient/s, or believe that you may have > received this communication in error, please reply to the sender indicating > that fact and delete the copy you received and in addition, you should not > print, copy, re-transmit, disseminate, or otherwise use the information > contained in this communication. Internet communications cannot be > guaranteed to be timely, secure, error or virus-free. The sender does not > accept liability for any errors or omissions. > -- Thanks, Miyuru Dayarathna Senior Technical Lead Mobile: +94713527783 <+94%2071%20352%207783> Blog: http://miyurublog.blogspot.com ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] Fwd: GSOC2016: [ML][CEP] [SAMOA]Predictive analytic with online data for WSO2 Machine Learner-Samoa Integration
Adding Jayan to this email thread. -- Thanks, Miyuru Dayarathna Senior Technical Lead Mobile: +94713527783 Blog: http://miyurublog.blogspot.com On Wed, Aug 3, 2016 at 3:11 PM, Supun Sethunga <sup...@wso2.com> wrote: > Hi Mahesh, > > you should build carbon-ml *without *-U option. -U means you force mvn to > look for updates in remote repo. Rather run it with -o option. Also, can > you double check whether the dependencies are defined correctly (group > Id's, versions etc). > > Alternatively, it seems there is a samoa released version in mvn repo. > Maybe you could try that one as well. But that's v0.3.0.. > > [1] https://mvnrepository.com/artifact/org.apache.samoa > > On Wed, Aug 3, 2016 at 2:54 PM, Mahesh Dananjaya < > dananjayamah...@gmail.com> wrote: > >> Hi Supun, >> I just neeed a little help. I am in the process of integrating my samoa >> core functions and extension into carbon-ml's siddhi extension. For samoa i >> am using locally built samoa project to provide samoa 0.4.0 >> dependencies,since we dont have it in the maven repo or else where. But >> when i build carbon-ml by adding samoa dependencies, it seems to be maven >> search for remote location, not the local maven repo first. I am running >> maven with -U option. But still the problems occurs. Is there any specific >> thing in carbon-ml like settings to search remote before local one? I just >> need to give local maven repo for the dependency. My extension seperately >> working fine, so there is no problem wihat the local dependencies outside >> carbon-ml. So can you please help me with this.thank you. >> regards, >> Mahesh. >> >> On Fri, Jul 22, 2016 at 3:17 PM, Srinath Perera <srin...@wso2.com> wrote: >> >>> Hi Mahesh, >>> >>> On Thu, Jul 21, 2016 at 2:10 PM, Mahesh Dananjaya < >>> dananjayamah...@gmail.com> wrote: >>> >>>> Hi All, >>>> I am onto connecting cep streams with samoa streams to data analysis >>>> using samoa framework. To connect samoa with cep siddhi event streams what >>>> i we can do is that try to convert cep streams into samoa streams or else >>>> writing wrpper for samoa for cep streasm to be used. In both cases i have >>>> to covert siddhi cep streasm into samoa streams. Samoa is using MOA to >>>> analyse data. Moo contains ML framework to analyse stream data. Samoa is >>>> wrapping MOA withsome of its classes. >>>> >>>> Samoa streams is based on MOA, Instance and InstanceStreams. Samoa see >>>> streams as a stream of instances [1]. So if we are going to convert cep >>>> events into samoa instances , it will take time. But if we have some >>>> similarity between cep siddhi streams and samoa streasm we can reduce the >>>> time. >>>> 1. What is the underlying infrastructure for cep siddhi streasm.? >>>> 2. Are there anything as Instances or InstanceStreams kind of >>>> implmentation underlying cep streams? >>>> 3. How can i get more underestanding on CEP siddhi streams. >>>> >>>> On the other hand i can use my cep siddhi extension and put those >>>> events into event queue and convert them into samoa instances and feed them >>>> into samoa streaming ml topologies. >>>> >>> I think this is OK. I assume this is much easier. Let's do this and >>> check the performance. >>> >>> >>>> There is another option. In Samoa what they are basically doing is that >>>> wrapping MOA ML framework and write some classes for build streaming ml >>>> topologies. So as the other option i can wrap samoa moa with my design and >>>> use moa ml framework directly. (No need for Samoa extension). I have >>>> building some topologies to streaming data analysis [2]. Main problem is >>>> that lack of documentation. Anyway i had go through their whole samoa >>>> design.thank you. >>>> >>> >>> If we use MOA directly, would we loose the distributed support in SAOMA. >>> Let's do a call when you can, so we can dsicuss this in detail. >>> >>> --Srinath >>> >>> >>> >>>> regards, >>>> Mahesh. >>>> >>>> [1] >>>> https://github.com/apache/incubator-samoa/blob/master/samoa-api/src/main/java/org/apache/samoa/streams/clustering/ClusteringStream.java >>>> [2] >>>> https://github.com/dananjayamahesh/GSOC2016/tree/master/gsoc/samoa/streaming/src/main/java/org/gsoc/samoa/streaming >>>
Re: [Dev] [APIM] Inclusion of available APIs under Deleted APIs list in API Deployment Stats in Log Analyzer
Adding Isabelle and Srinath to the email thread. -- Thanks, Miyuru Dayarathna Senior Technical Lead Mobile: +94713527783 Blog: http://miyurublog.blogspot.com On Tue, Jul 5, 2016 at 1:11 PM, Sajith Perera <saji...@wso2.com> wrote: > > Hi Maneesha, > > Log analyzer purely based on the log events which are generated by a > particular product. For each stat, there are set of keywords and rules in > log lines, which we are identifying as unique for that scenario. > > Regarding the above mentioned scenario, if there is an update on an API in > given time frame, there will be two log lines printed with "Destroying" and > "Initiating" keywords. > > 1. Deleted Artifact X > 2. Created Artifact X > > These events are listed as "Deployed Api X" and "Deleted Api X" tables in > the given time frame, as they are the "facts" within the given time frame. > > Thanks, > SajithD > > On Tue, Jul 5, 2016 at 12:44 PM, Maneesha Wijesekara <manee...@wso2.com> > wrote: > >> Hi all, >> >> Currently 'API Deployment Stats' in Log Analyzer listed both Deployed >> APIs and Deleted APIs. But I've noticed that some available APIs (not yet >> deleted) are also listed under 'Deleted APIs' [1]. The reason behind this >> is when an API is edited, it will destroyed and again re-deployed. That log >> will cause to include an available API under 'Deleted APIs'. But when it >> comes to the function 'Deleted APIs', it should only list the names of APIs >> which are actually deleted. >> >> Any thoughts on this ? >> >> [1] https://wso2.org/jira/browse/ANLYAPIM-79 >> >> -- >> Maneesha Wijesekara >> Software Engineer - QA Team >> WSO2 Inc. >> >> Email: manee...@wso2.com >> Linkedin: http://linkedin.com/in/maneeshawijesekara >> Mobile: +94712443119 >> >> ___________ >> Dev mailing list >> Dev@wso2.org >> http://wso2.org/cgi-bin/mailman/listinfo/dev >> >> > > > -- > Sajith Dimal > Software Engineer > Mobile : +94783101496 > WSO2 Inc. | http://wso2.com > lean.enterprise.middleware > -- Thanks, Miyuru Dayarathna Senior Technical Lead Mobile: +94713527783 Blog: http://miyurublog.blogspot.com ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] Regarding GSoC Project : Auto-tuning of WSO2 CEP
Hi Ramindu, Thanks for your interest on working on GSoC'16 proposal 17 "Auto-tuning of WSO2 CEP". Based on the GSoC 2016 timeline as the next step of this process please prepare a project proposal. The guidelines on preparation of the proposal are available in [1]. Two sample project proposals are on [2][3]. Please maintain the proposal in a public place such as in your blog. Please feel free to email us if you have any unclear points in preparing the project proposal. [1] http://write.flossmanuals.net/gsocstudentguide/writing-a-proposal/ [2] http://write.flossmanuals.net/gsocstudentguide/proposal-example-1/ [3] http://write.flossmanuals.net/gsocstudentguide/proposal-example-2/ Thanks, Miyuru Dayarathna, Ph.D. Senior Technical Lead, WSO2 Inc. Mobile: +94713527783 On Tue, Mar 8, 2016 at 10:02 PM, Ramindu Senarath < ramindu.senar...@gmail.com> wrote: > Hi Miyuru & Sriskandarajah, > > I am a undergraduate of University of Moratuwa, Faculty of Information > Technology. I am interested in contribute to this project. For a start I > have studied about CEP and did some work with it. Then I have read given > references and had some knowledge about project deliverables. Since I have > some knowledge about CEP, I am going to start dig into the source code. I > have build the source code for product-cep which I have cloned from git. > Could you please suggest a further step to start the project. > > *Thanks & Regards* > *Ramindu Senarath* > *Undergraduate* > *Faculty of Information Technology* > *University of Moratuwa* > *Mobile:** +94713639896* <%2B94713639896> > -- Thanks, Miyuru Dayarathna Senior Technical Lead Mobile: +94713527783 Blog: http://miyurublog.blogspot.com ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] WSO2 IS Logs
Hi Nadeesha, Yes, scripts created for SSO and oauth for long running tests are fine. Could you please share those with us? -- Thanks, Miyuru Dayarathna Senior Technical Lead Mobile: +94713527783 Blog: http://miyurublog.blogspot.com On Thu, Jan 28, 2016 at 2:40 PM, Srinath Perera <srin...@wso2.com> wrote: > We need all logs, and yes mainly wso2carbon.log. > > --Srinath > > On Thu, Jan 28, 2016 at 2:31 PM, Nadeesha Meegoda <nadees...@wso2.com> > wrote: > >> Hi Miyuru, >> >> Are you expecting wso2carbon.log as logs or what kind of specific logs >> are you looking for? We don't have logs to be shared for tests we ran on >> local environments cause those are deleted by now. But we may be able to >> share logs of our long running tests. We have scripts created for SSO and >> oauth for long running tests done for IS 5.1.0. Are you looking for those >> kind of scripts? >> >> Thanks, >> Nadeesha >> >> On Thu, Jan 28, 2016 at 2:23 PM, Srinath Perera <srin...@wso2.com> wrote: >> >>> >>> >>> On Thu, Jan 28, 2016 at 2:01 PM, Miyuru Dayarathna <miyu...@wso2.com> >>> wrote: >>> >>>> Hi, >>>> >>>> We are currently working on to apply WSO2 LogAnalyzer to conduct >>>> analytics on multiple WSO2 products. I want to try out WSO2 IS logs with >>>> LogAnalyzer. Could you please let me know whether you already have such >>>> logs created in your test environment which I could get a copy? Also do you >>>> have some test script(s) which you use to test WSO2 IS? >>>> >>>> -- >>>> Thanks, >>>> Miyuru Dayarathna >>>> Senior Technical Lead >>>> Mobile: +94713527783 >>>> Blog: http://miyurublog.blogspot.com >>>> >>> >>> >>> >>> -- >>> >>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>> Site: http://people.apache.org/~hemapani/ >>> Photos: http://www.flickr.com/photos/hemapani/ >>> Phone: 0772360902 >>> >> >> >> >> -- >> *Nadeesha Meegoda* >> Software Engineer - QA >> WSO2 Inc.; http://wso2.com >> lean.enterprise.middleware >> email : nadees...@wso2.com >> mobile: +94783639540 >> <%2B94%2077%202273555> >> > > > > -- > > Blog: http://srinathsview.blogspot.com twitter:@srinath_perera > Site: http://people.apache.org/~hemapani/ > Photos: http://www.flickr.com/photos/hemapani/ > Phone: 0772360902 > -- Thanks, Miyuru Dayarathna Senior Technical Lead Mobile: +94713527783 Blog: http://miyurublog.blogspot.com ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] API Manager Logs
Hi Pubudu, Thanks for sharing the log files and the Jmeter scripts. Ok, I will let you know if I have any further questions. -- Thanks, Miyuru Dayarathna Senior Technical Lead Mobile: +94713527783 Blog: http://miyurublog.blogspot.com On Thu, Jan 28, 2016 at 2:14 PM, Pubudu Priyashan <pubu...@wso2.com> wrote: > Hi Miyuru, > > Please find the requested log files and the Jmeter scripts attached > herewith. You should be able to run the Jmeter scripts on a API-M 1.10.0 > standalone server out of the box. All the pre-conditions/instructions for > each script are included in the README file. > > Additionally, the attached log files are from long running tests being > executed on the distributed setup explained below. Do let me know if you > have any further questions. > > > > > Cheers, > Pubudu. > > > > > > Pubudu D.P > Senior Software Engineer - QA Team | WSO2 inc. > Mobile : +94775464547 > > On Thu, Jan 28, 2016 at 1:46 PM, Miyuru Dayarathna <miyu...@wso2.com> > wrote: > >> Hi Pubudu, >> >> I need to use some logs from API Manager (APIM) for some experiments >> which uses WSO2 LogAnalyzer. I talked to NuwanD and he mentioned me that >> you are currently running some long-running test cases with APIM. Would it >> be possible for me to obtain the logs from those APIM instances? >> Furthermore, it would be great if you could share with me the test >> script(s) which you ran on APIM. I can try different the parameters of the >> script and generate some interesting scenarios and try to detect them using >> the LogAnalyzer. >> >> -- >> Thanks, >> Miyuru Dayarathna >> Senior Technical Lead >> Mobile: +94713527783 >> Blog: http://miyurublog.blogspot.com >> > > -- Thanks, Miyuru Dayarathna Senior Technical Lead Mobile: +94713527783 Blog: http://miyurublog.blogspot.com ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] WSO2 IS Logs
Hi Pandula, Thanks for sharing the scripts and the log files. I completed downloading all the log files. -- Thanks, Miyuru Dayarathna Senior Technical Lead Mobile: +94713527783 Blog: http://miyurublog.blogspot.com On Thu, Jan 28, 2016 at 2:58 PM, Pandula Kariyawasam <pand...@wso2.com> wrote: > Hi Miyuru, > > Please find the attached scripts for SAML SSO and OAuth. > > You can download some logs from following location. > http://192.168.18.74:8000/ > > Thanks, > Pandula > > > On Thu, Jan 28, 2016 at 2:48 PM, Miyuru Dayarathna <miyu...@wso2.com> > wrote: > >> Hi Nadeesha, >> >> Yes, scripts created for SSO and oauth for long running tests are fine. >> Could you please share those with us? >> >> -- >> Thanks, >> Miyuru Dayarathna >> Senior Technical Lead >> Mobile: +94713527783 >> Blog: http://miyurublog.blogspot.com >> >> >> >> On Thu, Jan 28, 2016 at 2:40 PM, Srinath Perera <srin...@wso2.com> wrote: >> >>> We need all logs, and yes mainly wso2carbon.log. >>> >>> --Srinath >>> >>> On Thu, Jan 28, 2016 at 2:31 PM, Nadeesha Meegoda <nadees...@wso2.com> >>> wrote: >>> >>>> Hi Miyuru, >>>> >>>> Are you expecting wso2carbon.log as logs or what kind of specific logs >>>> are you looking for? We don't have logs to be shared for tests we ran on >>>> local environments cause those are deleted by now. But we may be able to >>>> share logs of our long running tests. We have scripts created for SSO and >>>> oauth for long running tests done for IS 5.1.0. Are you looking for those >>>> kind of scripts? >>>> >>>> Thanks, >>>> Nadeesha >>>> >>>> On Thu, Jan 28, 2016 at 2:23 PM, Srinath Perera <srin...@wso2.com> >>>> wrote: >>>> >>>>> >>>>> >>>>> On Thu, Jan 28, 2016 at 2:01 PM, Miyuru Dayarathna <miyu...@wso2.com> >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> We are currently working on to apply WSO2 LogAnalyzer to conduct >>>>>> analytics on multiple WSO2 products. I want to try out WSO2 IS logs with >>>>>> LogAnalyzer. Could you please let me know whether you already have such >>>>>> logs created in your test environment which I could get a copy? Also do >>>>>> you >>>>>> have some test script(s) which you use to test WSO2 IS? >>>>>> >>>>>> -- >>>>>> Thanks, >>>>>> Miyuru Dayarathna >>>>>> Senior Technical Lead >>>>>> Mobile: +94713527783 >>>>>> Blog: http://miyurublog.blogspot.com >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>>>> Site: http://people.apache.org/~hemapani/ >>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>> Phone: 0772360902 >>>>> >>>> >>>> >>>> >>>> -- >>>> *Nadeesha Meegoda* >>>> Software Engineer - QA >>>> WSO2 Inc.; http://wso2.com >>>> lean.enterprise.middleware >>>> email : nadees...@wso2.com >>>> mobile: +94783639540 >>>> <%2B94%2077%202273555> >>>> >>> >>> >>> >>> -- >>> >>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>> Site: http://people.apache.org/~hemapani/ >>> Photos: http://www.flickr.com/photos/hemapani/ >>> Phone: 0772360902 >>> >> >> >> >> -- >> Thanks, >> Miyuru Dayarathna >> Senior Technical Lead >> Mobile: +94713527783 >> Blog: http://miyurublog.blogspot.com >> > > > > -- > > *Pandula Kariyawasam K.B.* > *Senior Software Engineer - QA* > Mobile: +94772314510 > > > > *WSO2 Inc.lean . enterprise . middlewear.http://www.wso2.com > <http://www.wso2.com>* > <http://www.wso2.com> > -- Thanks, Miyuru Dayarathna Senior Technical Lead Mobile: +94713527783 Blog: http://miyurublog.blogspot.com ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] CEP over Out-of-Order Data Streams
Hi Suho, Today's afternoon I merged the out-of-order handler (OOH) code to Siddhi and created a pull request. It comprises of a simple test case [2 <https://github.com/miyurud/siddhi/blob/master/modules/siddhi-extensions/ooh/src/test/java/org/wso2/siddhi/extension/ooh/KSlackExtensionTestCase.java>] which is described in [3 <https://drive.google.com/a/wso2.com/file/d/0B8UIlB9H3rv_THpoRi0xTFFERVE/view?usp=sharing>]. I tested the code with DEBS 2015 Grand Challenge's taxi trips data set [1 <http://www.debs2015.org/call-grand-challenge.html>] as well. I used the small 2 million events data set in the tests. Since the events in the original data set were in sorted order, I introduced a delay to certain events in the event stream before sending them to the OOH (Shown in green color highlighting in [4 <https://docs.google.com/a/wso2.com/spreadsheets/d/1xYGJoNfCSQTv1KsAuPNYsSSFqOEpn8gqCZy35L8GDmo/edit?usp=sharing>]). The results are listed in [4 <https://docs.google.com/a/wso2.com/spreadsheets/d/1xYGJoNfCSQTv1KsAuPNYsSSFqOEpn8gqCZy35L8GDmo/edit?usp=sharing>]. The OOH reorders event streams in both example scenarios except for few glitches. For 100 events case there was one unordered event output by OOH (Highlighted in yellow color in [4 <https://docs.google.com/a/wso2.com/spreadsheets/d/1xYGJoNfCSQTv1KsAuPNYsSSFqOEpn8gqCZy35L8GDmo/edit?usp=sharing>]). I am investigating solutions for such unordered events output by OOH. However, a similar situation has been described in Figure 2 of [3 <https://drive.google.com/a/wso2.com/file/d/0B8UIlB9H3rv_THpoRi0xTFFERVE/view?usp=sharing>] as well. Another notable observation is that considerable amount of events were buffered in the OOH while the extension was running. In certain cases (as shown in [4 <https://docs.google.com/a/wso2.com/spreadsheets/d/1xYGJoNfCSQTv1KsAuPNYsSSFqOEpn8gqCZy35L8GDmo/edit?usp=sharing>]) there were about 41 events buffered in OOH. Hence, both the test cases ended up outputting lesser number of events than what is input to OOH because the remaining set of events were buffered inside the OOH when the test case terminated. This is expected behavior of a buffer based out-of-order handler. Next, I will continue experiments with full data sets of DEBS 2013 and 2015 data sets and observe the characteristics of OOH buffer for different out-of-order scenarios. [1] http://www.debs2015.org/call-grand-challenge.html [2] https://github.com/miyurud/siddhi/blob/master/modules/siddhi-extensions/ooh/src/test/java/org/wso2/siddhi/extension/ooh/KSlackExtensionTestCase.java [3] https://drive.google.com/a/wso2.com/file/d/0B8UIlB9H3rv_THpoRi0xTFFERVE/view?usp=sharing [4] https://docs.google.com/a/wso2.com/spreadsheets/d/1xYGJoNfCSQTv1KsAuPNYsSSFqOEpn8gqCZy35L8GDmo/edit?usp=sharing -- Thanks, Miyuru Dayarathna Senior Technical Lead Mobile: +94713527783 Blog: http://miyurublog.blogspot.com On Wed, Sep 16, 2015 at 11:10 PM, Sriskandarajah Suhothayan <s...@wso2.com> wrote: > Hi Miyuru > > Any update on this ? > > Suho > > On Tue, Sep 8, 2015 at 7:58 PM, Miyuru Dayarathna <miyu...@wso2.com> > wrote: > >> Hi Suho, >> >> Ok, I will work on that. >> >> -- >> Thanks, >> Miyuru Dayarathna >> Senior Technical Lead >> Mobile: +94713527783 >> Blog: http://miyurublog.blogspot.com >> >> On Tue, Sep 8, 2015 at 5:38 PM, Sriskandarajah Suhothayan <s...@wso2.com> >> wrote: >> >>> Can you merge this code to siddhi and send a pull we will be able to add >>> this to the release >>> >>> Suho >>> >>> On Tue, Sep 8, 2015 at 5:17 PM, Miyuru Dayarathna <miyu...@wso2.com> >>> wrote: >>> >>>> Hi Suho, >>>> >>>> I have completed developing the K-Slack extension for Siddhi and I have >>>> hosted the code in [1 <https://github.com/miyurud/Siddhi_Dev>]. I have >>>> tested the KSlackExtension class [2 >>>> <https://github.com/miyurud/Siddhi_Dev/blob/master/modules/siddhi-extensions/ooh/src/main/java/org/wso2/siddhi/extension/ooh/KSlackExtension.java>] >>>> using a test scenario described in [3 >>>> <https://drive.google.com/a/wso2.com/file/d/0B8UIlB9H3rv_THpoRi0xTFFERVE/view?usp=sharing>] >>>> and I have verified the correctness of the implementation. The test case is >>>> accessible from [4 >>>> <https://github.com/miyurud/Siddhi_Dev/blob/master/modules/siddhi-extensions/ooh/src/test/java/org/wso2/siddhi/extension/ooh/KSlackExtensionTestCase.java>]. >>>> We need to discuss whether we should keep the following Siddhi syntax or >>>> change it to some different version by removing the use of "window." >>>> construct. This code assumes that the ve
Re: [Dev] CEP over Out-of-Order Data Streams
Hi Suho, I fixed all the issues mentioned during the code review and pushed the modifications to GitHub. -- Thanks, Miyuru Dayarathna Senior Technical Lead Mobile: +94713527783 Blog: http://miyurublog.blogspot.com On Thu, Sep 17, 2015 at 10:20 PM, Sriskandarajah Suhothayan <s...@wso2.com> wrote: > Thanks for the pull request there are some comments on the code please fix > them > > Thanks > Suho > > On Thu, Sep 17, 2015 at 4:56 PM, Miyuru Dayarathna <miyu...@wso2.com> > wrote: > >> Hi Suho, >> >> Today's afternoon I merged the out-of-order handler (OOH) code to Siddhi >> and created a pull request. It comprises of a simple test case [2 >> <https://github.com/miyurud/siddhi/blob/master/modules/siddhi-extensions/ooh/src/test/java/org/wso2/siddhi/extension/ooh/KSlackExtensionTestCase.java>] >> which is described in [3 >> <https://drive.google.com/a/wso2.com/file/d/0B8UIlB9H3rv_THpoRi0xTFFERVE/view?usp=sharing>]. >> I tested the code with DEBS 2015 Grand Challenge's taxi trips data set [1 >> <http://www.debs2015.org/call-grand-challenge.html>] as well. I used the >> small 2 million events data set in the tests. Since the events in the >> original data set were in sorted order, I introduced a delay to certain >> events in the event stream before sending them to the OOH (Shown in green >> color highlighting in [4 >> <https://docs.google.com/a/wso2.com/spreadsheets/d/1xYGJoNfCSQTv1KsAuPNYsSSFqOEpn8gqCZy35L8GDmo/edit?usp=sharing>]). >> The results are listed in [4 >> <https://docs.google.com/a/wso2.com/spreadsheets/d/1xYGJoNfCSQTv1KsAuPNYsSSFqOEpn8gqCZy35L8GDmo/edit?usp=sharing>]. >> The OOH reorders event streams in both example scenarios except for few >> glitches. For 100 events case there was one unordered event output by OOH >> (Highlighted in yellow color in [4 >> <https://docs.google.com/a/wso2.com/spreadsheets/d/1xYGJoNfCSQTv1KsAuPNYsSSFqOEpn8gqCZy35L8GDmo/edit?usp=sharing>]). >> I am investigating solutions for such unordered events output by OOH. >> However, a similar situation has been described in Figure 2 of [3 >> <https://drive.google.com/a/wso2.com/file/d/0B8UIlB9H3rv_THpoRi0xTFFERVE/view?usp=sharing>] >> as well. Another notable observation is that considerable amount of events >> were buffered in the OOH while the extension was running. In certain cases >> (as shown in [4 >> <https://docs.google.com/a/wso2.com/spreadsheets/d/1xYGJoNfCSQTv1KsAuPNYsSSFqOEpn8gqCZy35L8GDmo/edit?usp=sharing>]) >> there were about 41 events buffered in OOH. Hence, both the test cases >> ended up outputting lesser number of events than what is input to OOH >> because the remaining set of events were buffered inside the OOH when the >> test case terminated. This is expected behavior of a buffer based >> out-of-order handler. Next, I will continue experiments with full data sets >> of DEBS 2013 and 2015 data sets and observe the characteristics of OOH >> buffer for different out-of-order scenarios. >> >> [1] http://www.debs2015.org/call-grand-challenge.html >> >> [2] >> https://github.com/miyurud/siddhi/blob/master/modules/siddhi-extensions/ooh/src/test/java/org/wso2/siddhi/extension/ooh/KSlackExtensionTestCase.java >> >> [3] >> https://drive.google.com/a/wso2.com/file/d/0B8UIlB9H3rv_THpoRi0xTFFERVE/view?usp=sharing >> >> [4] >> https://docs.google.com/a/wso2.com/spreadsheets/d/1xYGJoNfCSQTv1KsAuPNYsSSFqOEpn8gqCZy35L8GDmo/edit?usp=sharing >> >> -- >> Thanks, >> Miyuru Dayarathna >> Senior Technical Lead >> Mobile: +94713527783 >> Blog: http://miyurublog.blogspot.com >> >> On Wed, Sep 16, 2015 at 11:10 PM, Sriskandarajah Suhothayan < >> s...@wso2.com> wrote: >> >>> Hi Miyuru >>> >>> Any update on this ? >>> >>> Suho >>> >>> On Tue, Sep 8, 2015 at 7:58 PM, Miyuru Dayarathna <miyu...@wso2.com> >>> wrote: >>> >>>> Hi Suho, >>>> >>>> Ok, I will work on that. >>>> >>>> -- >>>> Thanks, >>>> Miyuru Dayarathna >>>> Senior Technical Lead >>>> Mobile: +94713527783 >>>> Blog: http://miyurublog.blogspot.com >>>> >>>> On Tue, Sep 8, 2015 at 5:38 PM, Sriskandarajah Suhothayan < >>>> s...@wso2.com> wrote: >>>> >>>>> Can you merge this code to siddhi and send a pull we will be able to >>>>> add this to the release >>>>> >>>>> Suho >>>>> >>>>> On Tue, Sep 8, 2015
