Hi Maninda, On 24 October 2014 18:16, Maninda Edirisooriya <mani...@wso2.com> wrote:
> > On Fri, Oct 24, 2014 at 3:13 PM, Lasantha Fernando <lasan...@wso2.com> > wrote: > >> Hi Maninda, >> >> On 24 October 2014 14:40, Maninda Edirisooriya <mani...@wso2.com> wrote: >> >>> >>> On Fri, Oct 24, 2014 at 1:10 PM, Lasantha Fernando <lasan...@wso2.com> >>> wrote: >>> >>>> Hi Maninda, all, >>>> >>>> On 24 October 2014 12:39, Gihan Anuruddha <gi...@wso2.com> wrote: >>>> >>>>> We need to maintain a single location to give publisher credential >>>>> details. At the moment we are maintaining same details in a different >>>>> location that is a bit inconvenient. As an example if we change BAM >>>>> url then we have to change BAM server profiles as well as mediation stat >>>>> publish UI as well. Also, this password should support secureVault. >>>>> >>>> >>>> +1 >>>> >>> Yes. When we can come up with the generic artifact which describes data >>> sink URL and credentials, we can access them for all ESB components (like >>> new mediator and mediation stats publisher) via a declarative service. As >>> this artifact is dep synced across all the products other products too can >>> get URL and credentials without any issue. All you have to do is to install >>> the feature to that product that exposes that declarative service. >>> >>>> >>>> >>>>> >>>>> On Fri, Oct 24, 2014 at 12:25 PM, Maninda Edirisooriya < >>>>> mani...@wso2.com> wrote: >>>>> >>>>>> Hi all, >>>>>> >>>>>> Today morning we had a discussion where Anjana, Lahiru, Vijitha, >>>>>> Kasun and I participated. >>>>>> >>>>>> There we decided to develop a common Data Publisher / Data Sink axis2 >>>>>> artifact which is generic throughout the platform. This will enable any >>>>>> other carbon products also for finding data publishing connection / >>>>>> credential details when they want to publish data to CEP / BAM. >>>>>> >>>>> >>>> Are we going to couple with Axis2 for the data publisher/ data sink? >>>> Querying about this since you had mentioned 'axis2 artifacts'.. :-). >>>> >>> I am not sure about the new artifact type that is going to replace axis2 >>> artifact deployment, and that is the reason I mentioned the name "axis2". >>> Yes it is better if we can go with the new deployment artifact type. :) >>> >> >> In the current data-bridge model, we are not having Axis2 deployment >> artifacts since the data receivers/data publishers are not hot-deployable >> artifacts. In the current model, usually, we configure a single data >> receiver for a server before start up and the no. of publishers , type are >> also pre-configured and not hot-deployable. Are we going for a model with >> multiple data publisher/data receiver artifacts deployed through an Axis2 >> deployer or similar? >> > > You are correct. We can use a configuration file for configuring Data > Publishing server connection URL and credentials assuming all the tenants > are publishing to the same BAM / CEP server in a multi-tenant environment. > I think we can go for hot-deployable data sink/data publisher artifacts if there is a use case for that. I was actually wrong earlier in saying that data publishers are not hot-deployable. In BAM/CEP output adaptors, OutputWSO2Event adaptors are actually hot deployable. For ESB also, I think there is the ability to configure the BAM server profiles dynamically. Though I think the BAM server profiles are stored in registry and not hot deployable. If we move these configs to a file, we are actually removing existing functionality IMHO. So I guess we cannot move all the configs to a file-based config. +1 from me to make Data Publisher configurations hot deployable axis2 artifacts (Removing axis2 dependency has a long way to go yet, I think). Sorry for suggesting otherwise earlier.. :-) In databridge level, the receiver details and the no. of threads,event queue size are file-based configurations. The data sink/ data receiver artifacts were not made hot deployable with assumption that keeping a pre-configured single port listening for incoming Thrift traffic is adequate. Shall we make it the same for the new Data Publisher/ Data Sink configurations? i.e. file-based DataReceiver/DataSink and hot-deployable DataPublisher (We also need to get the names right as well, I think. i.e. DataPublisher vs EventPublihser, DataReceiver vs DataSink etc...) ? We can make the DataSink part hot-deployable as well if there is a use case for that? WDYT? > >> >>>> Can't we make it generic to the platform without coupling with Axis2 >>>> since we are moving away from Axis2 in other areas of the platform? >>>> >>> Can you explain how to make it generic? This is really a problem I >>> thought. I think we have to decide it before we completely get rid of >>> Axis2. Maybe Azeez have an idea. >>> >>> Azeez, what do you think about the new artifact deployment modal >>> independent from Axis2? >>> >>>> >>>> >>>>>> And specific to the ESB we are going to develop a mediator (it's name >>>>>> is not confirmed yet) which will be similar to the current BAM mediator >>>>>> which will refer to the above mentioned artifact for credentials and >>>>>> connection parameters. As also we should include the transport we are >>>>>> referring in that particular data sink / data publisher. For the first >>>>>> cut >>>>>> we can use only the Thrift and hope to extend for HTTP and JMS in future. >>>>>> >>>>>> In new implementation of the mediator, there will be no configuration >>>>>> for Stream configuration like the BAM mediator. Instead there will be >>>>>> inline configurations in the mediator XML which consists of >>>>>> 1. a stream ID (or the combination of Stream Name and Stream Version) >>>>>> 2. and mappings from message XPath to the stream field. (i.e. maps >>>>>> from message properties to the stream fields.) >>>>>> >>>>> >>>> +1 for mapping message to stream attributes as described above. >>>> >>>> >>>>>> For the convenience to the user, we can prompt the stream fields for >>>>>> mapping from the UI, (once stream ID is provided) if the stream is >>>>>> already >>>>>> there in the stream definition store. Anyway there (if want to get UI >>>>>> help) >>>>>> we have to use the existing streams that were already created from CEP or >>>>>> BAM. But still you can add a mapping for a new stream which can >>>>>> subsequently be added to the stream definition store from CEP or BAM. >>>>>> >>>>>> We have decided to go for a mediator instead of a connector as this >>>>>> has nothing to do for accessing an external API kind of operation but to >>>>>> publish data for our own products according to the connection information >>>>>> given by the internal Data Publisher / Data Sink artifact. >>>>>> >>>>>> WDYT about the cases when we use other transports like HTTP, JMS, >>>>>> MQTT etc. ? >>>>>> >>>>> >>>> Will we really be needing HTTP, JMS support when communicating within >>>> the platform across different products? Asking this since we already have >>>> similar functionality in many places (e.g. ESB, BAM/CEP input adaptors) and >>>> our general architectural approach is to get the events/messages into the >>>> platform from one product, then communicate within the platform using a >>>> single native format for efficiency/performance. WDYT? >>>> >>> >>> Due to the client loadbalancing requirement of the Thrift endpoint there >>> may be some limitations when it comes to publishing from one cluster to >>> another cluster where data receiver thrift URLs are not visible outside >>> from the second cluster. There we may have to use HTTP data sink type. And >>> if there is a requirement for publishing to a 3rd party JMS topic we may >>> need JMS support and so on. Anyway we should first focus on the Thrift type >>> and will think about others as the next steps. :) >>> >> >> If we are trying to send something over HTTP or JMS from ESB, can't we >> make use of the already existing implementations within ESB to send to HTTP >> or JMS endpoints? It will indeed be very useful to have a set of pluggable >> transports that can be used by any product of the platform. I think the >> concern is whether we will be redoing already available transports when >> implementing HTTP/JMS for databridge level as well. Of course, I might have >> misunderstood the approach mentioned here, if so, please correct me if am >> wrong.. :-) >> > In this data sink artifact we will build the messages according to the > events used in the stream where the existing HTTP, JMS and etc. message > endpoints are capable of sending raw messages. The example is Entitlement > mediator where it sends data via HTTP and with a specific format. > Yes, that makes sense. In that case, I think the data publisher/event publisher artifacts would provide a sort of pre-configured template for sending events over HTTP, JMS to other WSO2 products and use the underlying HTTP, JMS transport implementation that already exists. Thanks, Lasantha >> Thanks, >> Lasantha >> >> >>> >>>> Thanks, >>>> Lasantha >>>> >>>>> >>>>>> Feedback is welcome. >>>>>> >>>>>> Thanks. >>>>>> >>>>>> >>>>>> *Maninda Edirisooriya* >>>>>> Senior Software Engineer >>>>>> >>>>>> *WSO2, Inc.*lean.enterprise.middleware. >>>>>> >>>>>> *Blog* : http://maninda.blogspot.com/ >>>>>> *E-mail* : mani...@wso2.com >>>>>> *Skype* : @manindae >>>>>> *Twitter* : @maninda >>>>>> >>>>>> On Mon, Oct 20, 2014 at 8:00 PM, Sriskandarajah Suhothayan < >>>>>> s...@wso2.com> wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> On Mon, Oct 20, 2014 at 4:06 AM, Mohanadarshan Vivekanandalingam < >>>>>>> mo...@wso2.com> wrote: >>>>>>> >>>>>>>> Hi All, >>>>>>>> >>>>>>>> We had a discussion sometime back (last year) on this.. Please >>>>>>>> check the mail thread [1] for more info.. >>>>>>>> >>>>>>>> >>>>>>> Yes, I believe we all agree that we need to improve the current BAM >>>>>>> Mediator. >>>>>>> When I had a chat with kasun on this he suggested writing it as a >>>>>>> connector will be much cleaner. >>>>>>> >>>>>>> Where we can have the connection information and stream definition >>>>>>> in <init> and <defineStream> then then pass the data via <publish>. >>>>>>> >>>>>>> I believe its better to have a f2f meeting with kasun on this. >>>>>>> >>>>>>> Regards >>>>>>> Suho >>>>>>> >>>>>>> [1] [Architecture] Improving Data Publisher Mediator to replace BAM >>>>>>>> Mediator >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Mohan >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Oct 20, 2014 at 12:55 PM, Anjana Fernando <anj...@wso2.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi Sanjiva, >>>>>>>>> >>>>>>>>> On Sat, Oct 18, 2014 at 9:44 PM, Sanjiva Weerawarana < >>>>>>>>> sanj...@wso2.com> wrote: >>>>>>>>> >>>>>>>>>> On Fri, Oct 17, 2014 at 6:47 PM, Lahiru Chandima < >>>>>>>>>> lahi...@wso2.com> wrote: >>>>>>>>>> >>>>>>>>>>> Hi all, >>>>>>>>>>> >>>>>>>>>>> We (Vijitha<vijit...@wso2.com> and myself) are going to develop >>>>>>>>>>> an ESB Connector which can publish events to BAM and CEP. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> We're currently using the words "ESB Connector" to mean something >>>>>>>>>> that becomes a set of mediators to access (primarily) HTTP APIs. >>>>>>>>>> This is >>>>>>>>>> not that right?? If so lets not use the same words please. >>>>>>>>>> >>>>>>>>> >>>>>>>>> Yeah, I guess, this is simply a mediator, just like what we have >>>>>>>>> now. It was just Kasun has mentioned to Suho, to create this as an ESB >>>>>>>>> Connector, where we haven't actually gone into what it means to >>>>>>>>> create a >>>>>>>>> connector. @Kasun, maybe you can shed some light on this, on what you >>>>>>>>> meant >>>>>>>>> earlier. >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Currently, BAM mediator is used for sending events from ESB to >>>>>>>>>>> BAM. BAM Mediator needs to be configured with server details and >>>>>>>>>>> streams >>>>>>>>>>> need to be defined prior to using them in sequences of ESB Synapse >>>>>>>>>>> configuration. >>>>>>>>>>> >>>>>>>>>>> With new ESB connector, it will be possible to add all >>>>>>>>>>> configurations needed to publish events to BAM or CEP in the ESB >>>>>>>>>>> Synapse >>>>>>>>>>> configuration itself. This brings the configuration to a single >>>>>>>>>>> place >>>>>>>>>>> unlike when using the BAM mediator. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Is this a real requirement? Do we expect different ESB configs >>>>>>>>>> (running on the same server) to want to send data to different >>>>>>>>>> BAM/CEP >>>>>>>>>> endpoints? I understand the multitenancy usecase - is it only for >>>>>>>>>> that? >>>>>>>>>> >>>>>>>>> >>>>>>>>> Actually, the idea here is, what we have now for defining the >>>>>>>>> target for events, "BAM Profiles", to replace that. Basically in a BAM >>>>>>>>> Profile, we put all the information such as the event receiver >>>>>>>>> details, and >>>>>>>>> the stream definition, and how the properties are mapped using >>>>>>>>> properties >>>>>>>>> from a sequence. We want to clean this up by, simply having something >>>>>>>>> equivalent to a message store we have now, where we named is "data >>>>>>>>> sink" >>>>>>>>> for now, to be the target for events, and the actual stream mapping >>>>>>>>> will be >>>>>>>>> done by the mediator itself, in the sequence. And the motivation for >>>>>>>>> having >>>>>>>>> this as an artifact in the synapse configuration is, where from a >>>>>>>>> place >>>>>>>>> like DevStudio or any development environment, we can create these >>>>>>>>> deployment artifacts can directly deploy them, without going to the >>>>>>>>> admin >>>>>>>>> console and adding them through the UI. >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> *High level architecture - components.* >>>>>>>>>>> >>>>>>>>>>> 1) New Synapse construct will be introduced to configure details >>>>>>>>>>> related to the data sink (ie. CEP or BAM). >>>>>>>>>>> >>>>>>>>>>> Eg. >>>>>>>>>>> >>>>>>>>>>> <dataSink name=”bam_thrift_server”> >>>>>>>>>>> <url>tcp://localhost:7611<url> >>>>>>>>>>> <username>admin</username> >>>>>>>>>>> <password>admin</password> >>>>>>>>>>> </dataSink> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> What do we call this now? The term "dataSink" doesn't work for me >>>>>>>>>> but that could be just me. >>>>>>>>>> >>>>>>>>> >>>>>>>>> Yeah, we had a meeting sometime back to discuss this, and that day >>>>>>>>> also, we couldn't decide on the name :) .. basically something >>>>>>>>> related to a >>>>>>>>> dataSink, any suggestions? :) .. >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> 2) An ESB connector will be developed which can be used to >>>>>>>>>>> publish events to a data sink. Connector will use data-bridge API >>>>>>>>>>> (Thrift) >>>>>>>>>>> for publishing events (It will be possible to introduce new >>>>>>>>>>> transports >>>>>>>>>>> other than Thrift later). >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Once again, what are the operations? Why do we want to write a >>>>>>>>>> connector instead of using / improving the current "BAM mediator"? >>>>>>>>>> >>>>>>>>> >>>>>>>>> So yeah, it will basically be improving/re-writing the current >>>>>>>>> mediator, with a name like "dataPublish" or something, without any >>>>>>>>> "BAM" >>>>>>>>> part, and with the stream property mapping part added there itself, >>>>>>>>> rather >>>>>>>>> than having it in another place. >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Thrift stream related data (stream name, version and stream >>>>>>>>>>> properties) will be configured in the ESB connector configuration >>>>>>>>>>> (in >>>>>>>>>>> Synapse config), which is more logical compared to configuring it >>>>>>>>>>> separately as done in the BAM mediator. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Can you give a specific example to illustrate the difference >>>>>>>>>> please? >>>>>>>>>> >>>>>>>>> >>>>>>>>> So basically in the current BAM mediator, we do not give the >>>>>>>>> direct mapping to the stream properties there, we simply populate some >>>>>>>>> message context properties and call the mediator. And the mediator >>>>>>>>> simply >>>>>>>>> looks up the information provided by the BAM Profile to do the actual >>>>>>>>> mapping, where in our profile, we have given like Xpath expressions >>>>>>>>> to say, >>>>>>>>> what are the values in a specific target stream event. >>>>>>>>> >>>>>>>>> Cheers, >>>>>>>>> Anjana. >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Sanjiva. >>>>>>>>>> -- >>>>>>>>>> Sanjiva Weerawarana, Ph.D. >>>>>>>>>> Founder, Chairman & CEO; WSO2, Inc.; http://wso2.com/ >>>>>>>>>> email: sanj...@wso2.com; office: (+1 650 745 4499 | +94 11 214 >>>>>>>>>> 5345) x5700; cell: +94 77 787 6880 | +1 408 466 5099; voip: +1 >>>>>>>>>> 650 265 8311 >>>>>>>>>> blog: http://sanjiva.weerawarana.org/; twitter: @sanjiva >>>>>>>>>> Lean . Enterprise . Middleware >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Architecture mailing list >>>>>>>>>> Architecture@wso2.org >>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> *Anjana Fernando* >>>>>>>>> Senior Technical Lead >>>>>>>>> WSO2 Inc. | http://wso2.com >>>>>>>>> lean . enterprise . middleware >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Architecture mailing list >>>>>>>>> Architecture@wso2.org >>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> *V. Mohanadarshan* >>>>>>>> *Software Engineer,* >>>>>>>> *Data Technologies Team,* >>>>>>>> *WSO2, Inc. http://wso2.com <http://wso2.com> * >>>>>>>> *lean.enterprise.middleware.* >>>>>>>> >>>>>>>> email: mo...@wso2.com >>>>>>>> phone:(+94) 771117673 >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> *S. Suhothayan* >>>>>>> Technical Lead & Team Lead of WSO2 Complex Event Processor >>>>>>> *WSO2 Inc. *http://wso2.com >>>>>>> * <http://wso2.com/>* >>>>>>> lean . enterprise . middleware >>>>>>> >>>>>>> >>>>>>> *cell: (+94) 779 756 757 <%28%2B94%29%20779%20756%20757> | blog: >>>>>>> http://suhothayan.blogspot.com/ >>>>>>> <http://suhothayan.blogspot.com/>twitter: >>>>>>> http://twitter.com/suhothayan <http://twitter.com/suhothayan> | >>>>>>> linked-in: >>>>>>> http://lk.linkedin.com/in/suhothayan >>>>>>> <http://lk.linkedin.com/in/suhothayan>* >>>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Architecture mailing list >>>>>> Architecture@wso2.org >>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> W.G. Gihan Anuruddha >>>>> Senior Software Engineer | WSO2, Inc. >>>>> M: +94772272595 >>>>> >>>>> _______________________________________________ >>>>> Architecture mailing list >>>>> Architecture@wso2.org >>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>> >>>>> >>>> >>>> >>>> -- >>>> *Lasantha Fernando* >>>> Software Engineer - Data Technologies Team >>>> WSO2 Inc. http://wso2.com >>>> >>>> email: lasan...@wso2.com >>>> mobile: (+94) 71 5247551 >>>> >>>> _______________________________________________ >>>> Architecture mailing list >>>> Architecture@wso2.org >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> >>>> >>> >>> _______________________________________________ >>> Architecture mailing list >>> Architecture@wso2.org >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> *Lasantha Fernando* >> Software Engineer - Data Technologies Team >> WSO2 Inc. http://wso2.com >> >> email: lasan...@wso2.com >> mobile: (+94) 71 5247551 >> >> _______________________________________________ >> Architecture mailing list >> Architecture@wso2.org >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > -- *Lasantha Fernando* Software Engineer - Data Technologies Team WSO2 Inc. http://wso2.com email: lasan...@wso2.com mobile: (+94) 71 5247551
_______________________________________________ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
