On 15 August 2013 13:08, Maninda Edirisooriya <[email protected]> wrote:
> On Wed, Aug 14, 2013 at 4:23 PM, Sriskandarajah Suhothayan > <[email protected]>wrote: > >> >> >> >> On Wed, Aug 14, 2013 at 3:12 PM, Maninda Edirisooriya >> <[email protected]>wrote: >> >>> >>> Hi Suho, >>> >>> On Wed, Aug 14, 2013 at 1:28 PM, Sriskandarajah Suhothayan < >>> [email protected]> wrote: >>> >>>> >>>> +1 for this approach >>>> The other solution is BAM listening to Databridge Stream Definitions >>>> and creating the necessary Event builder configurations problematically. >>>> Through this all the streams will come through the >>>> InputTranportAdaptors and EventBuilders >>>> >>> I also have considered about that and it is great if we can go with this >>> solution. But there is a problem I think. Suppose this case. >>> 1. Very first event comes to the stream. Then our Databridge listener >>> will detect the new stream and create a new Event builder. >>> 2. While the new Event builder is created and deployed the second event >>> of the new stream will arrive. >>> 3. Second message will get lost. >>> >> >> When BAM listener can deploy Event builders for all available stream at >> deployment time and for the runtime stream defining case, since the define >> stream is synchronous (as we have discussed offline )the actual event will >> arrive only after the Event builder gets deployed. >> > +1. I clarified this with Suho off-line. This solution will work for the > requirement. For the information of others let me explain the new solution. > 1. BAM listener will be listening to the Databridge for new streams. > 2. When a new stream comes it will be notified to our listener from > Databridge. > 3. Listener will check the stream name and version with the allowed > streams configuration (mentioned in the 1st mail) > 4. If matches, create a new Event builder and deploy it. > 5. Add a listener to the new Event builder for messages. > 6. When a message arrives, the Event builder will send it to the listener. > 7. The listener will store the message in Cassandra. > +1. This approach looks more clean IMO than the earlier one. Thanks, Lasantha > >> Suho >> >> >> >>> If our Event builder creation is fast enough so that it will not be that >>> late to drop the second message, we can use the given approach. >>> >>>> >>>> Suho >>>> >>>> >>>> On Wed, Aug 14, 2013 at 1:07 PM, Maninda Edirisooriya <[email protected] >>>> > wrote: >>>> >>>>> At the moment in BAM, all the data received from data bridge are >>>>> directly stored to the Cassandra. When a new stream creation message is >>>>> received, that stream is automatically created and starts to listen to >>>>> that >>>>> stream automatically. >>>>> >>>>> As a new feature we are going to re-use the InputTransportAdapter and >>>>> InputMessageBuilder (currently used by CEP) to receive messages to BAM. >>>>> The >>>>> rationale is to unify the message receiving features in both CEP and BAM >>>>> which will enable BAM to receive message from other transports such as >>>>> JMS. >>>>> >>>>> One problem with this new feature is enabling the automatic stream >>>>> creation with InputTransportAdapter and InputMessageBuilder. The user has >>>>> to pre-configure the streams it is intended to be listened so that new >>>>> message builders are created for each stream. >>>>> >>>>> The second problem is filtering the streams come to CEP and BAM. Some >>>>> streams may be only required to be used by CEP but not BAM. The reason is >>>>> keeping all the messages comes to CEP in Cassandra will waste the disk >>>>> space. But there is no stream filtering mechanism either in CEP or BAM at >>>>> the moment. >>>>> >>>>> The solution we have come up is as follows. >>>>> >>>>> 1. BAM is listening to both event builders and data bridge. >>>>> 2. Event builders should be pre configured if the streams have to be >>>>> received via event builders. >>>>> 3. But still all the streams come to data bridge will be received by >>>>> the data bridge listener. >>>>> 4. When a new stream creation message is received either to event >>>>> builder or to data bridge, they will notify it to our new lisetner. >>>>> 5. There is a separate configuration stating what are the streams >>>>> accepted by the BAM. If that configuration is "*", all the streams should >>>>> be accepted. (should go to Cassandra) If there is a comma seperated list >>>>> of >>>>> streams, only those streams should go to Cassandra. If a set of streams >>>>> stated after a "!" symbol, all the streams except those streams should go >>>>> to Cassandra. >>>>> 6. When a stream creation message is notified to a listener, that >>>>> listener should create a stream in the Cassandra only if that stream is >>>>> matching with the stream set stated in the config file. >>>>> 7. Now the problem is receiving the same message from both event >>>>> builder and the data bridge. Therefore when a stream creation is notified, >>>>> the notifier (either event builder or the data bridge) is remembered by >>>>> the >>>>> listener. Only that notifier will be allowed to store events to the >>>>> Cassandra using that stream. In other words, either event builder or the >>>>> data bridge can store events using a unique stream. >>>>> >>>>> >>>>> * >>>>> Maninda Edirisooriya* >>>>> Software Engineer >>>>> *WSO2, Inc. >>>>> *lean.enterprise.middleware. >>>>> >>>>> *Blog* : http://maninda.blogspot.com/ >>>>> *Phone* : +94 777603226 >>>>> >>>>> _______________________________________________ >>>>> Architecture mailing list >>>>> [email protected] >>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>> >>>>> >>>> >>>> >>>> -- >>>> *S. Suhothayan >>>> * >>>> Associate Technical Lead, >>>> *WSO2 Inc. *http://wso2.com * >>>> <http://wso2.com/>* >>>> lean . enterprise . middleware >>>> >>>> *cell: (+94) 779 756 757 | blog: http://suhothayan.blogspot.com/ >>>> twitter: http://twitter.com/suhothayan | linked-in: >>>> http://lk.linkedin.com/in/suhothayan* >>>> * >>>> * >>>> >>>> _______________________________________________ >>>> Architecture mailing list >>>> [email protected] >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> >>>> >>> >> >> >> -- >> *S. Suhothayan >> * >> Associate Technical Lead, >> *WSO2 Inc. *http://wso2.com * >> <http://wso2.com/>* >> lean . enterprise . middleware >> >> *cell: (+94) 779 756 757 | blog: http://suhothayan.blogspot.com/ >> twitter: http://twitter.com/suhothayan | linked-in: >> http://lk.linkedin.com/in/suhothayan* >> * >> * >> > > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- *Lasantha Fernando* Software Engineer - Data Technologies Team WSO2 Inc. http://wso2.com email: [email protected] mobile: (+94) 71 5247551
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
