I'd like to add some requirements in if possible. 1) I'd like some really easy dynamic stuff. For example, if I'm using REST, can I just send a JSON? We could create a stream definition dynamically with the name based on some part of the URL and compare to previous definitions for a version number. This isn't going to be efficient, but in many cases I just want to get on with things.
I think we have to be aware we have three potentially different usecases here: a) High performance BAM and CEP - need the stream defined in advance for efficiency b) Ad hoc BAM and logging - maybe we don't want to be so prescriptive / statically defined c) Lightweight CEP / prototyping - being able to use a stream like a dynamic language and do duck typing would be useful. 2) I'd like to support both a Registry model but also being able to point to a stream definition JSON via a URL. This would make this nice and "webby"/ REST-friendly. 3) We need to make it super easy to define a stream, edit stream definitions, find them, etc. Paul On 20 January 2014 11:10, Srinath Perera <srin...@wso2.com> wrote: > Hi All, > > I am reviving this thread as now we have support to add remove streams > though CEP UI (and BAM?) and via toolboxes, but we still define the event > streams with the data publisher each time. (this thread discusses about > stop doing that). > > Typical CEP/ BAM usecase would go like following > > 1) first you identify and define streams > 2) then you write queries > 3) then you write/configure the agents and point to streams they should > publish to. > > However, we redefine streams from agents every time we publish. This has > several complications. > > 1) If an agent choose to define stream different from queries, everything > get messed up. (problem was in samples in BAM). > 2) If there are multiple event receivers, we need a way to sync them with > db everytime we update a stream. > > I would like to propose we switch to a model where we write queries and > streams and deploy them as a toolbox. Then, just mention the stream name in > the event publisher to keep things simple. > > --Srinath > > > > On Sat, Jan 19, 2013 at 5:41 PM, Tharindu Mathew <thari...@wso2.com>wrote: > >> A store discover already supported event streams seems to be a good idea. >> It would be good idea which we can implement through a UI or a store. >> >> But, what do you mean by defining event streams everywhere? >> >> Clients can define any stream they want, but it is only defined at the >> stream definition store. If there is an error, it is shown in the stream >> definition store and returned to the client (I hope it is returned at >> least, otherwise it is a bug). >> >> This model was first proposed to get out of the eventing hell that we put >> ourselves into. Right now, if you want to publish something, you just >> define it and publish. You don't have to switch between multiple servers >> just to define and publish some events, which was extremely annoying. >> >> >> >> >> On Sat, Jan 19, 2013 at 2:28 PM, Amila Suriarachchi <am...@wso2.com>wrote: >> >>> >>> >>> On Fri, Jan 18, 2013 at 7:54 PM, Sriskandarajah Suhothayan < >>> s...@wso2.com> wrote: >>> >>>> >>>> >>>> On Fri, Jan 18, 2013 at 6:46 PM, Sanjiva Weerawarana >>>> <sanj...@wso2.com>wrote: >>>> >>>>> +1 for the stream definition store idea (a new use of the Store). >>>>> >>>>> +1 >>>> >>>>> Suho can we change our client API to take the streamID as a param? >>>>> That way a user can look it up in the store and use it directly. >>>>> >>>>> Currently we do have server side APIs to get StreamDefintons from >>>> StreamId or StreamName & Version. >>>> >>> >>> I think we need to separate out the event definition from runtime event >>> publishing. >>> >>> If we look at how brokers used in CEP, first users need to define broker >>> at the broker Manager and use the broker id at the CEP bucket level. For >>> event streams we can have default set of streams (which BAM publishers and >>> other default Agents use) and let users to define at run time. At the event >>> publishing side they can use the stream id. >>> >>> thanks, >>> Amila. >>> >>> >>>> >>>> Sanjiva. >>>>> >>>>> >>>>> On Fri, Jan 18, 2013 at 6:33 PM, Sriskandarajah Suhothayan < >>>>> s...@wso2.com> wrote: >>>>> >>>>>> >>>>>> >>>>>> On Fri, Jan 18, 2013 at 4:23 PM, Srinath Perera <srin...@wso2.com>wrote: >>>>>> >>>>>>> Hi Suho, >>>>>>> >>>>>>> Yes, users can define them. But they do not always have to, and we >>>>>>> should recommend that users go and do that once rather than doing it >>>>>>> every >>>>>>> time. >>>>>>> >>>>>> Yes this is possible even now. First time when we define stream we >>>>>> get a streamId and then we can simply reuse that (even after restart). >>>>>> The issue is, our clients are not capable to store the streamId >>>>>> returned by the server, and therefore to get the already defined streamId >>>>>> corresponding to the stream definition we are redefining the stream. >>>>>> This is because our streamId is not simply regeneratable like >>>>>> <streamName>-<version> but we are using <streamName>-<version>-UUID. >>>>>> >>>>>> Suho >>>>>> >>>>>>> >>>>>>> --Srinath >>>>>>> >>>>>>> >>>>>>> On Thu, Jan 17, 2013 at 10:22 PM, Sriskandarajah Suhothayan < >>>>>>> s...@wso2.com> wrote: >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Fri, Jan 18, 2013 at 2:00 AM, Srinath Perera >>>>>>>> <srin...@wso2.com>wrote: >>>>>>>> >>>>>>>>> Currently we handle CEP streams by defining it everywhere we >>>>>>>>> publish >>>>>>>>> events, and ignoring the duplicates. However, if content of events >>>>>>>>> change, then we throw an exception. >>>>>>>>> >>>>>>>>> I have concerns that we are creating a data stream definition hell >>>>>>>>> by >>>>>>>>> defining it every where. Also, when something changes, you would >>>>>>>>> have >>>>>>>>> to change things everywhere. >>>>>>>>> >>>>>>>>> Maybe we should just put that via registry in one place >>>>>>>>> >>>>>>>>> e.g. Go to registry and add the definitions. When you publish, >>>>>>>>> just >>>>>>>>> use event stream name only, and do not redefine events. >>>>>>>>> >>>>>>>>> +1 to make registry based Stream Definition Store. >>>>>>>> but we still need to allow the clients to define streams. >>>>>>>> We have the ability to delete stream so I believe that is good >>>>>>>> enough to. >>>>>>>> The disadvantage in Cassandra based Stream Definition Store is that >>>>>>>> it dont have a UI and hence we cannot tweak the stream Definition. >>>>>>>> IMHO having UI for the Stream Definition Store will be a solution. >>>>>>>> >>>>>>>> Suho >>>>>>>> >>>>>>>> --Srinath >>>>>>>>> >>>>>>>>> -- >>>>>>>>> ============================ >>>>>>>>> Srinath Perera, Ph.D. >>>>>>>>> Senior Software Architect, WSO2 Inc. >>>>>>>>> Visiting Faculty, University of Moratuwa >>>>>>>>> Member, Apache Software Foundation >>>>>>>>> Research Scientist, Lanka Software Foundation >>>>>>>>> Blog: http://srinathsview.blogspot.com/ >>>>>>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>>>>>> Phone: 0772360902 >>>>>>>>> _______________________________________________ >>>>>>>>> Architecture mailing list >>>>>>>>> Architecture@wso2.org >>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> *S. Suhothayan* >>>>>>>> Software Engineer, >>>>>>>> Data Technologies Team, >>>>>>>> *WSO2, Inc. * >>>>>>>> *http://wso2.com <http://wso2.com/> <http://wso2.com/>* >>>>>>>> *lean.enterprise.middleware.* >>>>>>>> >>>>>>>> *email: **s...@wso2.com* <s...@wso2.com> >>>>>>>> * cell: (+94) 779 756 757 blog: >>>>>>>> **http://suhothayan.blogspot.com/*<http://suhothayan.blogspot.com/> >>>>>>>> *twitter: >>>>>>>> **http://twitter.com/suhothayan*<http://twitter.com/suhothayan> >>>>>>>> * linked-in: **http://lk.linkedin.com/in/suhothayan >>>>>>>> <http://lk.linkedin.com/in/suhothayan>* >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> ============================ >>>>>>> Srinath Perera, Ph.D. >>>>>>> http://www.cs.indiana.edu/~hperera/ >>>>>>> http://srinathsview.blogspot.com/ >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> *S. Suhothayan* >>>>>> Software Engineer, >>>>>> Data Technologies Team, >>>>>> *WSO2, Inc. * >>>>>> *http://wso2.com <http://wso2.com/> <http://wso2.com/>* >>>>>> *lean.enterprise.middleware.* >>>>>> >>>>>> *email: **s...@wso2.com* <s...@wso2.com> >>>>>> * cell: (+94) 779 756 757 blog: >>>>>> **http://suhothayan.blogspot.com/*<http://suhothayan.blogspot.com/> >>>>>> *twitter: **http://twitter.com/suhothayan*<http://twitter.com/suhothayan> >>>>>> * linked-in: **http://lk.linkedin.com/in/suhothayan >>>>>> <http://lk.linkedin.com/in/suhothayan>* >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Architecture mailing list >>>>>> Architecture@wso2.org >>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Sanjiva Weerawarana, Ph.D. >>>>> Founder, Chairman & CEO; WSO2, Inc.; http://wso2.com/ >>>>> email: sanj...@wso2.com; phone: +94 11 763 9614; cell: +94 77 787 6880| +1 >>>>> 650 265 8311 >>>>> blog: http://sanjiva.weerawarana.org/ >>>>> >>>>> Lean . Enterprise . Middleware >>>>> >>>>> _______________________________________________ >>>>> Architecture mailing list >>>>> Architecture@wso2.org >>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>> >>>>> >>>> >>>> >>>> -- >>>> >>>> *S. Suhothayan* >>>> Software Engineer, >>>> Data Technologies Team, >>>> *WSO2, Inc. * >>>> *http://wso2.com <http://wso2.com/> <http://wso2.com/>* >>>> *lean.enterprise.middleware.* >>>> >>>> *email: **s...@wso2.com* <s...@wso2.com> >>>> * cell: (+94) 779 756 757 blog: >>>> **http://suhothayan.blogspot.com/*<http://suhothayan.blogspot.com/> >>>> *twitter: **http://twitter.com/suhothayan*<http://twitter.com/suhothayan> >>>> * linked-in: **http://lk.linkedin.com/in/suhothayan >>>> <http://lk.linkedin.com/in/suhothayan>* >>>> >>>> >>>> _______________________________________________ >>>> Architecture mailing list >>>> Architecture@wso2.org >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> >>>> >>> >>> >>> -- >>> *Amila Suriarachchi* >>> >>> Software Architect >>> WSO2 Inc. ; http://wso2.com >>> lean . enterprise . middleware >>> >>> phone : +94 71 3082805 >>> >>> _______________________________________________ >>> Architecture mailing list >>> Architecture@wso2.org >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> Regards, >> >> Tharindu Mathew >> >> Associate Technical Lead, WSO2 BAM >> Member - Data Mgmt. Committee >> >> blog: http://tharindumathew.com/ >> M: +94777759908 >> >> _______________________________________________ >> Architecture mailing list >> Architecture@wso2.org >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > ============================ > Srinath Perera, Ph.D. > http://people.apache.org/~hemapani/ > http://srinathsview.blogspot.com/ > > _______________________________________________ > Architecture mailing list > Architecture@wso2.org > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- Paul Fremantle CTO and Co-Founder, WSO2 OASIS WS-RX TC Co-chair, Apache Member UK: +44 207 096 0336 US: +1 646 595 7614 blog: http://pzf.fremantle.org twitter.com/pzfreo p...@wso2.com wso2.com Lean Enterprise Middleware Disclaimer: This communication may contain privileged or other confidential information and is intended exclusively for the addressee/s. If you are not the intended recipient/s, or believe that you may have received this communication in error, please reply to the sender indicating that fact and delete the copy you received and in addition, you should not print, copy, retransmit, disseminate, or otherwise use the information contained in this communication. Internet communications cannot be guaranteed to be timely, secure, error or virus-free. The sender does not accept liability for any errors or omissions.
_______________________________________________ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
