Here are some of my findings, 1.Using thrift to send the information to centralised CEP will be much efficient then using Hazelcast cluster, 2.In Siddhi since its open for users to write their own quires, if they write any with sliding time window the memory consumption will be high as CEP need to remember all the events that occurred during that time window, but if the query uses only batch windows (like every min/hour/day you are allowed x ) then it wont be a problem
Suho On Mon, Jun 10, 2013 at 1:51 PM, Paul Fremantle <p...@wso2.com> wrote: > Suho > > In this case, where the rules are pretty simple, would the CEP overhead be > large? > > Paul > > > On 10 June 2013 09:06, Sriskandarajah Suhothayan <s...@wso2.com> wrote: > >> In addition to what Paul suggested if we want to offload the CEP's memory >> conception and processing overhead from the clustered APIM gateway, we can >> send all the events to CEP via thrift (same as sending to BAM ) and the CEP >> will send the throttling events back when the throttling threshold is >> reached >> >> Suho >> >> >> On Mon, Jun 10, 2013 at 1:18 PM, Paul Fremantle <p...@wso2.com> wrote: >> >>> Isabelle >>> >>> We will need to have Siddhi talk to either a database or to Hazelcast. >>> For many cases, we will need some persistent store because there will be >>> rules such as up to 10k / month and we need to persist that. Siddhi already >>> has some DB access, as well as some sharing across a cluster with Hazelcast. >>> >>> Paul >>> >>> >>> On 10 June 2013 07:08, Isabelle Mauny <isabe...@wso2.com> wrote: >>> >>>> How will this work in a cluster ? >>>> Throttling decisions are typically taken using cluster-wide data ( such >>>> as 50 reqs for this API, no matter how many gateways are running ) . >>>> >>>> Isabelle. >>>> >>>> ------ >>>> Isabelle Mauny >>>> Director, Product Management; WSO2, Inc.; http://wso2.com/ >>>> email: isabe...@wso2.com <isabe...@wso2.com> - mobile: +34 616050684 >>>> >>>> >>>> On Thu, Jun 6, 2013 at 7:04 PM, Paul Fremantle <p...@wso2.com> wrote: >>>> >>>>> Nuwan >>>>> >>>>> Siddhi is just a Java library, so this can be embedded. >>>>> >>>>> Paul >>>>> >>>>> >>>>> On 6 June 2013 17:52, Nuwan Dias <nuw...@wso2.com> wrote: >>>>> >>>>>> On Thu, Jun 6, 2013 at 3:28 PM, Sriskandarajah Suhothayan < >>>>>> s...@wso2.com> wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Thu, Jun 6, 2013 at 3:10 PM, Vijayaratha Vijayasingam < >>>>>>> rat...@wso2.com> wrote: >>>>>>> >>>>>>>> Hi; >>>>>>>> APIM team has different kind of throttling+rate limiting >>>>>>>> requirements (eg:ip/timer/geo based)..As Srinath pointed we believe >>>>>>>> that >>>>>>>> CEP would be the right solution, since we cannot have all these >>>>>>>> different >>>>>>>> scenarios/requirements to be implemented in our throttling module.. >>>>>>>> Is there any possibility integrate CEP engine with our throttling >>>>>>>> module ? >>>>>>>> >>>>>>> >>>>>>> Yes this is possible >>>>>>> Currently you can use Siddhi directly to achieve this. In this case >>>>>>> you have to pass all the events the trigger throttling directly to >>>>>>> Siddhi >>>>>>> and, when a matching throttling condition reached CEP will fire an >>>>>>> output >>>>>>> back to APIM so that it can take action based on the CEP's response. >>>>>>> >>>>>> >>>>>> How feasible is it to run Siddhi as an embedded engine within APIM? I >>>>>> am thinking along the lines of avoiding an additional hop between the API >>>>>> Gateway and CEP. >>>>>> >>>>>> Is it also possible to persist the throttling information to a >>>>>> database? Probably not on every request but batch by batch >>>>>> >>>>>>> >>>>>>> With CEP 3.0.0 we are writing an Event Processor component( as of >>>>>>> mail thread "CEP 3.0 towards a Complete eventing solution to the >>>>>>> platform"), when that is done you will also be able to provide a proper >>>>>>> UI >>>>>>> for the users to edit Siddhi throttling queries >>>>>>> >>>>>>> >>>>>>> So, all products can still use throttling module, which can handle >>>>>>>> all the complex rate/thorrtling policies.. >>>>>>>> >>>>>>> >>>>>>> +1 >>>>>>> >>>>>>> >>>>>>> Suho >>>>>>> >>>>>>> >>>>>>> >>>>>>>> Thanks >>>>>>>> >>>>>>>> >>>>>>>> On 5 June 2013 15:13, Sriskandarajah Suhothayan <s...@wso2.com>wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, Jun 5, 2013 at 1:27 AM, Srinath Perera >>>>>>>>> <srin...@wso2.com>wrote: >>>>>>>>> >>>>>>>>>> Hi All, >>>>>>>>>> >>>>>>>>>> Currently we have a java based throttling solution. But we need >>>>>>>>>> that to extended (e.g. support throughput based throttling), and >>>>>>>>>> support >>>>>>>>>> more complicated condition that currently parameterized. >>>>>>>>>> >>>>>>>>>> IMO, best way to do this is to support this by integrating CEP >>>>>>>>>> (Siddhi) engine directly at java level. It is very light weight . We >>>>>>>>>> can >>>>>>>>>> let users provide CEP queries which will control throttling. >>>>>>>>>> Basically, >>>>>>>>>> there will be inbuilt event stream definitions, and Siddhi listener >>>>>>>>>> that >>>>>>>>>> monitors a given event stream and adjust event acceptance. Users >>>>>>>>>> provide >>>>>>>>>> CEP queries. >>>>>>>>>> >>>>>>>>>> I think it is too heavy publish events via thrift API if we try >>>>>>>>>> to send it via the network. >>>>>>>>>> >>>>>>>>>> WDYT? >>>>>>>>>> >>>>>>>>> +1 >>>>>>>>> CEP team can provide the necessary support >>>>>>>>> if any of the product teams (eg: ELB, BPS, AF or AM) is willing to >>>>>>>>> replace their current or have an alternate throttling module >>>>>>>>> >>>>>>>>> Suho >>>>>>>>> >>>>>>>>>> >>>>>>>>>> --Srinath >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> ============================ >>>>>>>>>> Srinath Perera, Ph.D. >>>>>>>>>> Director, Research, WSO2 Inc. >>>>>>>>>> Visiting Faculty, University of Moratuwa >>>>>>>>>> Member, Apache Software Foundation >>>>>>>>>> Research Scientist, Lanka Software Foundation >>>>>>>>>> Blog: http://srinathsview.blogspot.com/ >>>>>>>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>>>>>>> Phone: 0772360902 >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> *S. Suhothayan >>>>>>>>> * >>>>>>>>> Software Engineer, >>>>>>>>> Management Committee Member, Data Technologies Team, >>>>>>>>> *WSO2 Inc. *http://wso2.com * >>>>>>>>> <http://wso2.com/>* >>>>>>>>> lean . enterprise . middleware >>>>>>>>> >>>>>>>>> *cell: (+94) 779 756 757 | blog: http://suhothayan.blogspot.com/ >>>>>>>>> twitter: http://twitter.com/suhothayan | linked-in: >>>>>>>>> http://lk.linkedin.com/in/suhothayan* >>>>>>>>> * >>>>>>>>> * >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Architecture mailing list >>>>>>>>> Architecture@wso2.org >>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> -Ratha >>>>>>>> mobile: (+94)755906608 >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Architecture mailing list >>>>>>>> Architecture@wso2.org >>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *S. Suhothayan >>>>>>> * >>>>>>> Associate Technical Lead, >>>>>>> >>>>>>> Management Committee Member, Data Technologies Team, >>>>>>> *WSO2 Inc. *http://wso2.com * >>>>>>> <http://wso2.com/>* >>>>>>> lean . enterprise . middleware >>>>>>> >>>>>>> >>>>>>> *cell: (+94) 779 756 757 | blog: http://suhothayan.blogspot.com/ >>>>>>> twitter: http://twitter.com/suhothayan | linked-in: >>>>>>> http://lk.linkedin.com/in/suhothayan* >>>>>>> * >>>>>>> * >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Architecture mailing list >>>>>>> Architecture@wso2.org >>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Nuwan Dias >>>>>> >>>>>> Senior Software Engineer - WSO2, Inc. http://wso2.com >>>>>> email : nuw...@wso2.com >>>>>> Phone : +94 777 775 729 >>>>>> >>>>>> _______________________________________________ >>>>>> Architecture mailing list >>>>>> Architecture@wso2.org >>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Paul Fremantle >>>>> CTO and Co-Founder, WSO2 >>>>> OASIS WS-RX TC Co-chair, VP, Apache Synapse >>>>> >>>>> UK: +44 207 096 0336 >>>>> US: +1 646 595 7614 >>>>> >>>>> blog: http://pzf.fremantle.org >>>>> twitter.com/pzfreo >>>>> p...@wso2.com >>>>> >>>>> wso2.com Lean Enterprise Middleware >>>>> >>>>> Disclaimer: This communication may contain privileged or other >>>>> confidential information and is intended exclusively for the addressee/s. >>>>> If you are not the intended recipient/s, or believe that you may have >>>>> received this communication in error, please reply to the sender >>>>> indicating >>>>> that fact and delete the copy you received and in addition, you should not >>>>> print, copy, retransmit, disseminate, or otherwise use the information >>>>> contained in this communication. Internet communications cannot be >>>>> guaranteed to be timely, secure, error or virus-free. The sender does not >>>>> accept liability for any errors or omissions. >>>>> >>>>> _______________________________________________ >>>>> Architecture mailing list >>>>> Architecture@wso2.org >>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>> >>>>> >>>> >>>> _______________________________________________ >>>> Architecture mailing list >>>> Architecture@wso2.org >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> >>>> >>> >>> >>> -- >>> Paul Fremantle >>> CTO and Co-Founder, WSO2 >>> OASIS WS-RX TC Co-chair, VP, Apache Synapse >>> >>> UK: +44 207 096 0336 >>> US: +1 646 595 7614 >>> >>> blog: http://pzf.fremantle.org >>> twitter.com/pzfreo >>> p...@wso2.com >>> >>> wso2.com Lean Enterprise Middleware >>> >>> Disclaimer: This communication may contain privileged or other >>> confidential information and is intended exclusively for the addressee/s. >>> If you are not the intended recipient/s, or believe that you may have >>> received this communication in error, please reply to the sender indicating >>> that fact and delete the copy you received and in addition, you should not >>> print, copy, retransmit, disseminate, or otherwise use the information >>> contained in this communication. Internet communications cannot be >>> guaranteed to be timely, secure, error or virus-free. The sender does not >>> accept liability for any errors or omissions. >>> >>> _______________________________________________ >>> Architecture mailing list >>> Architecture@wso2.org >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> *S. Suhothayan >> * >> Associate Technical Lead, >> Management Committee Member, Data Technologies Team, >> *WSO2 Inc. *http://wso2.com * >> <http://wso2.com/>* >> lean . enterprise . middleware >> >> *cell: (+94) 779 756 757 | blog: http://suhothayan.blogspot.com/ >> twitter: http://twitter.com/suhothayan | linked-in: >> http://lk.linkedin.com/in/suhothayan* >> * >> * >> >> _______________________________________________ >> Architecture mailing list >> Architecture@wso2.org >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > Paul Fremantle > CTO and Co-Founder, WSO2 > OASIS WS-RX TC Co-chair, VP, Apache Synapse > > UK: +44 207 096 0336 > US: +1 646 595 7614 > > blog: http://pzf.fremantle.org > twitter.com/pzfreo > p...@wso2.com > > wso2.com Lean Enterprise Middleware > > Disclaimer: This communication may contain privileged or other > confidential information and is intended exclusively for the addressee/s. > If you are not the intended recipient/s, or believe that you may have > received this communication in error, please reply to the sender indicating > that fact and delete the copy you received and in addition, you should not > print, copy, retransmit, disseminate, or otherwise use the information > contained in this communication. Internet communications cannot be > guaranteed to be timely, secure, error or virus-free. The sender does not > accept liability for any errors or omissions. > > _______________________________________________ > Architecture mailing list > Architecture@wso2.org > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- *S. Suhothayan * Associate Technical Lead, Management Committee Member, Data Technologies Team, *WSO2 Inc. *http://wso2.com * <http://wso2.com/>* lean . enterprise . middleware *cell: (+94) 779 756 757 | blog: http://suhothayan.blogspot.com/ twitter: http://twitter.com/suhothayan | linked-in: http://lk.linkedin.com/in/suhothayan* * *
_______________________________________________ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
