Hi All, I like to add some more information in to this..
On Mon, Jan 20, 2014 at 6:12 PM, Sinthuja Ragendran <sinth...@wso2.com>wrote: > Hi, > > > On Mon, Jan 20, 2014 at 5:36 PM, Maninda Edirisooriya <mani...@wso2.com>wrote: > >> Already BAM can create streams with REST API. To facilitate further for a >> person configuring BAM manually without a toolbox, we should provide a UI >> to create streams. >> > Yes Maninda, But there is a limitation where event needs to have a strict format.. But we are implementing a HTTP event adaptor which can be used to send any type of event (like json, xml and text) and do the processing at the CEP end (EventBuilder helps to convert these events into common format). > > CEP already has UI to create streams [1], I think it's for registry stream > defintion store. I'm not sure whether it'll support cassandra stream > definition store at the moment. We need to test and make it compatible with > Cassandra stream definition store as well for BAM. > No, At the moment CEP does not support cassandra as a stream store (only registry based), because we are directly storing the stream in registry using stream manger component. But we should use the databridge stream store component when dealing with stream. But we had some limitations in stream store component in CEP perspective but Now i think with the current re-factoring of databridge we can achieve this. And also we are going to improve the stream store UI to give more functionality (like editing, sample event generation and etc..).. I'll share more details in an another thread. > > > (and better if possible to facilitate edit and delete streams) And that UI >> should support defining indexes when required as current implementation of >> defining indexes is not nice. >> > > The event streams are common for BAM and CEP hence we should be careful > when adding indexes, etc to that common UI where it's not relevant to CEP > usecase. Here we have to load the attributes dynamically depending on what > stream definition store (cassandra, registry, etc), so that we can add the > relevant configuration like indexes, etc. > +1.. Thanks & Regards, Mohan > > [1] http://docs.wso2.org/display/CEP300/Working+with+Event+Streams > > Thanks, > Sinthuja. > >> >> As REST API fulfils light weight requirements of BAM, we can remove the >> Thrift based stream creation feature from the data publisher side. In high >> performance requirement of BAM, creating a stream from publisher side is >> not valid. They should deploy a toolbox at the beginning instead. Anyway >> this will make it backward incompatible for older BAM versions. >> >> And BTW what about keeping all the toolboxes in a WSO2 Store as discussed >> earlier in the thread? >> >> >> * Maninda Edirisooriya* >> Software Engineer >> >> *WSO2, Inc.*lean.enterprise.middleware. >> >> *Blog* : http://maninda.blogspot.com/ >> *Phone* : +94 777603226 >> >> >> On Mon, Jan 20, 2014 at 4:58 PM, Paul Fremantle <p...@wso2.com> wrote: >> >>> I'd like to add some requirements in if possible. >>> >>> 1) I'd like some really easy dynamic stuff. For example, if I'm using >>> REST, can I just send a JSON? We could create a stream definition >>> dynamically with the name based on some part of the URL and compare to >>> previous definitions for a version number. This isn't going to be >>> efficient, but in many cases I just want to get on with things. >>> >>> I think we have to be aware we have three potentially different usecases >>> here: >>> a) High performance BAM and CEP - need the stream defined in advance for >>> efficiency >>> b) Ad hoc BAM and logging - maybe we don't want to be so prescriptive / >>> statically defined >>> c) Lightweight CEP / prototyping - being able to use a stream like a >>> dynamic language and do duck typing would be useful. >>> >>> 2) I'd like to support both a Registry model but also being able to >>> point to a stream definition JSON via a URL. This would make this nice and >>> "webby"/ REST-friendly. >>> >>> 3) We need to make it super easy to define a stream, edit stream >>> definitions, find them, etc. >>> >>> Paul >>> >>> >>> On 20 January 2014 11:10, Srinath Perera <srin...@wso2.com> wrote: >>> >>>> Hi All, >>>> >>>> I am reviving this thread as now we have support to add remove streams >>>> though CEP UI (and BAM?) and via toolboxes, but we still define the event >>>> streams with the data publisher each time. (this thread discusses about >>>> stop doing that). >>>> >>>> Typical CEP/ BAM usecase would go like following >>>> >>>> 1) first you identify and define streams >>>> 2) then you write queries >>>> 3) then you write/configure the agents and point to streams they should >>>> publish to. >>>> >>>> However, we redefine streams from agents every time we publish. This >>>> has several complications. >>>> >>>> 1) If an agent choose to define stream different from queries, >>>> everything get messed up. (problem was in samples in BAM). >>>> 2) If there are multiple event receivers, we need a way to sync them >>>> with db everytime we update a stream. >>>> >>>> I would like to propose we switch to a model where we write queries and >>>> streams and deploy them as a toolbox. Then, just mention the stream name in >>>> the event publisher to keep things simple. >>>> >>>> --Srinath >>>> >>>> >>>> >>>> On Sat, Jan 19, 2013 at 5:41 PM, Tharindu Mathew <thari...@wso2.com>wrote: >>>> >>>>> A store discover already supported event streams seems to be a good >>>>> idea. It would be good idea which we can implement through a UI or a >>>>> store. >>>>> >>>>> But, what do you mean by defining event streams everywhere? >>>>> >>>>> Clients can define any stream they want, but it is only defined at the >>>>> stream definition store. If there is an error, it is shown in the stream >>>>> definition store and returned to the client (I hope it is returned at >>>>> least, otherwise it is a bug). >>>>> >>>>> This model was first proposed to get out of the eventing hell that we >>>>> put ourselves into. Right now, if you want to publish something, you just >>>>> define it and publish. You don't have to switch between multiple servers >>>>> just to define and publish some events, which was extremely annoying. >>>>> >>>>> >>>>> >>>>> >>>>> On Sat, Jan 19, 2013 at 2:28 PM, Amila Suriarachchi <am...@wso2.com>wrote: >>>>> >>>>>> >>>>>> >>>>>> On Fri, Jan 18, 2013 at 7:54 PM, Sriskandarajah Suhothayan < >>>>>> s...@wso2.com> wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> On Fri, Jan 18, 2013 at 6:46 PM, Sanjiva Weerawarana < >>>>>>> sanj...@wso2.com> wrote: >>>>>>> >>>>>>>> +1 for the stream definition store idea (a new use of the Store). >>>>>>>> >>>>>>>> +1 >>>>>>> >>>>>>>> Suho can we change our client API to take the streamID as a param? >>>>>>>> That way a user can look it up in the store and use it directly. >>>>>>>> >>>>>>>> Currently we do have server side APIs to get StreamDefintons from >>>>>>> StreamId or StreamName & Version. >>>>>>> >>>>>> >>>>>> I think we need to separate out the event definition from runtime >>>>>> event publishing. >>>>>> >>>>>> If we look at how brokers used in CEP, first users need to define >>>>>> broker at the broker Manager and use the broker id at the CEP bucket >>>>>> level. >>>>>> For event streams we can have default set of streams (which BAM >>>>>> publishers >>>>>> and other default Agents use) and let users to define at run time. At the >>>>>> event publishing side they can use the stream id. >>>>>> >>>>>> thanks, >>>>>> Amila. >>>>>> >>>>>> >>>>>>> >>>>>>> Sanjiva. >>>>>>>> >>>>>>>> >>>>>>>> On Fri, Jan 18, 2013 at 6:33 PM, Sriskandarajah Suhothayan < >>>>>>>> s...@wso2.com> wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Fri, Jan 18, 2013 at 4:23 PM, Srinath Perera >>>>>>>>> <srin...@wso2.com>wrote: >>>>>>>>> >>>>>>>>>> Hi Suho, >>>>>>>>>> >>>>>>>>>> Yes, users can define them. But they do not always have to, and >>>>>>>>>> we should recommend that users go and do that once rather than doing >>>>>>>>>> it >>>>>>>>>> every time. >>>>>>>>>> >>>>>>>>> Yes this is possible even now. First time when we define stream >>>>>>>>> we get a streamId and then we can simply reuse that (even after >>>>>>>>> restart). >>>>>>>>> The issue is, our clients are not capable to store the streamId >>>>>>>>> returned by the server, and therefore to get the already defined >>>>>>>>> streamId >>>>>>>>> corresponding to the stream definition we are redefining the stream. >>>>>>>>> This is because our streamId is not simply regeneratable like >>>>>>>>> <streamName>-<version> but we are using <streamName>-<version>-UUID. >>>>>>>>> >>>>>>>>> Suho >>>>>>>>> >>>>>>>>>> >>>>>>>>>> --Srinath >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Thu, Jan 17, 2013 at 10:22 PM, Sriskandarajah Suhothayan < >>>>>>>>>> s...@wso2.com> wrote: >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Fri, Jan 18, 2013 at 2:00 AM, Srinath Perera < >>>>>>>>>>> srin...@wso2.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Currently we handle CEP streams by defining it everywhere we >>>>>>>>>>>> publish >>>>>>>>>>>> events, and ignoring the duplicates. However, if content of >>>>>>>>>>>> events >>>>>>>>>>>> change, then we throw an exception. >>>>>>>>>>>> >>>>>>>>>>>> I have concerns that we are creating a data stream definition >>>>>>>>>>>> hell by >>>>>>>>>>>> defining it every where. Also, when something changes, you >>>>>>>>>>>> would have >>>>>>>>>>>> to change things everywhere. >>>>>>>>>>>> >>>>>>>>>>>> Maybe we should just put that via registry in one place >>>>>>>>>>>> >>>>>>>>>>>> e.g. Go to registry and add the definitions. When you publish, >>>>>>>>>>>> just >>>>>>>>>>>> use event stream name only, and do not redefine events. >>>>>>>>>>>> >>>>>>>>>>>> +1 to make registry based Stream Definition Store. >>>>>>>>>>> but we still need to allow the clients to define streams. >>>>>>>>>>> We have the ability to delete stream so I believe that is good >>>>>>>>>>> enough to. >>>>>>>>>>> The disadvantage in Cassandra based Stream Definition Store is >>>>>>>>>>> that it dont have a UI and hence we cannot tweak the stream >>>>>>>>>>> Definition. >>>>>>>>>>> IMHO having UI for the Stream Definition Store will be a >>>>>>>>>>> solution. >>>>>>>>>>> >>>>>>>>>>> Suho >>>>>>>>>>> >>>>>>>>>>> --Srinath >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> ============================ >>>>>>>>>>>> Srinath Perera, Ph.D. >>>>>>>>>>>> Senior Software Architect, WSO2 Inc. >>>>>>>>>>>> Visiting Faculty, University of Moratuwa >>>>>>>>>>>> Member, Apache Software Foundation >>>>>>>>>>>> Research Scientist, Lanka Software Foundation >>>>>>>>>>>> Blog: http://srinathsview.blogspot.com/ >>>>>>>>>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>>>>>>>>> Phone: 0772360902 >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>> Architecture@wso2.org >>>>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> >>>>>>>>>>> *S. Suhothayan* >>>>>>>>>>> Software Engineer, >>>>>>>>>>> Data Technologies Team, >>>>>>>>>>> *WSO2, Inc. * >>>>>>>>>>> *http://wso2.com <http://wso2.com/> <http://wso2.com/>* >>>>>>>>>>> *lean.enterprise.middleware.* >>>>>>>>>>> >>>>>>>>>>> *email: **s...@wso2.com* <s...@wso2.com> >>>>>>>>>>> * cell: (+94) 779 756 757 blog: * >>>>>>>>>>> *http://suhothayan.blogspot.com/*<http://suhothayan.blogspot.com/> >>>>>>>>>>> *twitter: >>>>>>>>>>> **http://twitter.com/suhothayan*<http://twitter.com/suhothayan> >>>>>>>>>>> * linked-in: **http://lk.linkedin.com/in/suhothayan >>>>>>>>>>> <http://lk.linkedin.com/in/suhothayan>* >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> ============================ >>>>>>>>>> Srinath Perera, Ph.D. >>>>>>>>>> http://www.cs.indiana.edu/~hperera/ >>>>>>>>>> http://srinathsview.blogspot.com/ >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> *S. Suhothayan* >>>>>>>>> Software Engineer, >>>>>>>>> Data Technologies Team, >>>>>>>>> *WSO2, Inc. * >>>>>>>>> *http://wso2.com <http://wso2.com/> <http://wso2.com/>* >>>>>>>>> *lean.enterprise.middleware.* >>>>>>>>> >>>>>>>>> *email: **s...@wso2.com* <s...@wso2.com> >>>>>>>>> * cell: (+94) 779 756 757 blog: >>>>>>>>> **http://suhothayan.blogspot.com/*<http://suhothayan.blogspot.com/> >>>>>>>>> *twitter: >>>>>>>>> **http://twitter.com/suhothayan*<http://twitter.com/suhothayan> >>>>>>>>> * linked-in: **http://lk.linkedin.com/in/suhothayan >>>>>>>>> <http://lk.linkedin.com/in/suhothayan>* >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Architecture mailing list >>>>>>>>> Architecture@wso2.org >>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Sanjiva Weerawarana, Ph.D. >>>>>>>> Founder, Chairman & CEO; WSO2, Inc.; http://wso2.com/ >>>>>>>> email: sanj...@wso2.com; phone: +94 11 763 9614; cell: +94 77 787 >>>>>>>> 6880 | +1 650 265 8311 >>>>>>>> blog: http://sanjiva.weerawarana.org/ >>>>>>>> >>>>>>>> Lean . Enterprise . Middleware >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Architecture mailing list >>>>>>>> Architecture@wso2.org >>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> *S. Suhothayan* >>>>>>> Software Engineer, >>>>>>> Data Technologies Team, >>>>>>> *WSO2, Inc. * >>>>>>> *http://wso2.com <http://wso2.com/> <http://wso2.com/>* >>>>>>> *lean.enterprise.middleware.* >>>>>>> >>>>>>> *email: **s...@wso2.com* <s...@wso2.com> >>>>>>> * cell: (+94) 779 756 757 blog: >>>>>>> **http://suhothayan.blogspot.com/*<http://suhothayan.blogspot.com/> >>>>>>> *twitter: >>>>>>> **http://twitter.com/suhothayan*<http://twitter.com/suhothayan> >>>>>>> * linked-in: **http://lk.linkedin.com/in/suhothayan >>>>>>> <http://lk.linkedin.com/in/suhothayan>* >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Architecture mailing list >>>>>>> Architecture@wso2.org >>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Amila Suriarachchi* >>>>>> >>>>>> Software Architect >>>>>> WSO2 Inc. ; http://wso2.com >>>>>> lean . enterprise . middleware >>>>>> >>>>>> phone : +94 71 3082805 >>>>>> >>>>>> _______________________________________________ >>>>>> Architecture mailing list >>>>>> Architecture@wso2.org >>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Regards, >>>>> >>>>> Tharindu Mathew >>>>> >>>>> Associate Technical Lead, WSO2 BAM >>>>> Member - Data Mgmt. Committee >>>>> >>>>> blog: http://tharindumathew.com/ >>>>> M: +94777759908 >>>>> >>>>> _______________________________________________ >>>>> Architecture mailing list >>>>> Architecture@wso2.org >>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>> >>>>> >>>> >>>> >>>> -- >>>> ============================ >>>> Srinath Perera, Ph.D. >>>> http://people.apache.org/~hemapani/ >>>> http://srinathsview.blogspot.com/ >>>> >>>> _______________________________________________ >>>> Architecture mailing list >>>> Architecture@wso2.org >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> >>>> >>> >>> >>> -- >>> Paul Fremantle >>> CTO and Co-Founder, WSO2 >>> OASIS WS-RX TC Co-chair, Apache Member >>> >>> UK: +44 207 096 0336 >>> US: +1 646 595 7614 >>> >>> blog: http://pzf.fremantle.org >>> twitter.com/pzfreo >>> p...@wso2.com >>> >>> wso2.com Lean Enterprise Middleware >>> >>> Disclaimer: This communication may contain privileged or other >>> confidential information and is intended exclusively for the addressee/s. >>> If you are not the intended recipient/s, or believe that you may have >>> received this communication in error, please reply to the sender indicating >>> that fact and delete the copy you received and in addition, you should not >>> print, copy, retransmit, disseminate, or otherwise use the information >>> contained in this communication. Internet communications cannot be >>> guaranteed to be timely, secure, error or virus-free. The sender does not >>> accept liability for any errors or omissions. >>> >>> _______________________________________________ >>> Architecture mailing list >>> Architecture@wso2.org >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> _______________________________________________ >> Architecture mailing list >> Architecture@wso2.org >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > *Sinthuja Rajendran* > Software Engineer <http://wso2.com/> > WSO2, Inc.:http://wso2.com > > Blog: http://sinthu-rajan.blogspot.com/ > Mobile: +94774273955 > > > > _______________________________________________ > Architecture mailing list > Architecture@wso2.org > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- *V. Mohanadarshan* *Software Engineer,* *Data Technologies Team,* *WSO2, Inc. http://wso2.com <http://wso2.com> * *lean.enterprise.middleware.* email: mo...@wso2.com phone:(+94) 771117673
_______________________________________________ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
