Hi Nathasha, According to the above graph, IMO Thrift Data Publisher is not asynchronous enough.
Are we currently invoking the data publisher in the same thread where the request is being processed? If so shall we try to move that logic to a separate thread? Thanks On Wed, Mar 2, 2016 at 2:51 PM, Nathasha Naranpanawa <natha...@wso2.com> wrote: > Hi, > > As initially planned, the feature was implemented to collect web > application statistics and publish them to DAS. > > Given below is an illustration of the high-level architecture of the > feature; > > > Currently, all configuration parameters required to create a connection to > DAS (such as DAS username, password, urls, etc.) are provided as valve > attributes in the server.xml file. The events are published through a > Thrift data agent and publishing is handled by the DataPublisher API which > is asynchronous. > > If DAS is not reachable, all the events will be buffered in order. When > DAS is made available, the buffered events will be pushed to DAS and > published. This is handled by the DataPublisher API itself. > > A performance testing was also carried out using Jmeter in order to > compare the throughput with and without the publisher valve. Following is > an indication of the throughput of the server according to a set of data > obtained; > > > > The Spark script which is currently in the Capp is a sample one. The Cron > expression in the script is scheduled to execute the script at every > second. The script presents two tables, raw statistics data and request > summary per minute. The table that holds raw statistics data are updated at > every invoke of a web application. The request summary per minute table is > only updated at each minute of invoke. Summarizing information published to > DAS and displaying them to users is done as a separate project by another > intern by creating gadgets to be deployed on the Dashboard server. > > Any further suggestions and feedback are highly appreciated. > > Thanks, > Nathasha. > > > On Tue, Dec 15, 2015 at 5:04 PM, Imesh Gunaratne <im...@wso2.com> wrote: > >> >> On Tue, Dec 15, 2015 at 11:29 AM, Nathasha Naranpanawa <natha...@wso2.com >>>> > wrote: >>>> >>>>> >>>>> - Adding a custom Valve to the tomcat server in order to collect >>>>> data from the server. >>>>> >>>>> Can you please explain more details about this valve? >> >> - What would be the approach taken to publish data in asynchronous >> mode? >> - How do we plan to share data between the thread that accept the >> request and the thread that publish statistics? >> - How would that scale/affect if the incoming requests grow? >> - How would this valve would handle if DAS is not reachable? >> >> Thanks >> >>> >>>>> - Creating necessary CApp packages in order to define means of >>>>> publishing data from tomcat server to DAS, receiving and storing the >>>>> data. >>>>> >>>>> The custom tomcat Valve behaves as an Event Publisher and publishes >>>>> data to DAS through Thrift data agent. For each request from the server, >>>>> an >>>>> Event is created including metadata and payload data and is published to >>>>> the DAS by a DataPublisher. >>>>> >>>>> A CApp was created by including the definitions for an Event Stream, >>>>> Event Receiver and Event Store to receive data from the publisher valve >>>>> and >>>>> store them. This is deployed in the >>>>> wso2das-3.0.0/repository/deployment/server/carbonapps folder. >>>>> >>>>> A Spark script is also added to the CApp in order to query the data >>>>> received by DAS. The script presents a summary of the requests received by >>>>> DAS per minute. The summary includes the following information; >>>>> >>>>> - average request count >>>>> - average response time >>>>> - HTTP request success count >>>>> - HTTP request error count >>>>> >>>>> Any suggestions and feedback are highly appreciated. >>>>> >>>>> Thanks, >>>>> >>>>> >>>>> -- >>>>> Nathasha Naranpanawa >>>>> Software Engineering Intern >>>>> WSO2 Inc. >>>>> >>>>> Email: natha...@wso2.com >>>>> Mobile: +94775496142 >>>>> LinkedIn: https://lk.linkedin.com/in/nathashanaranpanawa >>>>> >>>>> >>>>> _______________________________________________ >>>>> Dev mailing list >>>>> d...@wso2.org >>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>> >>>>> >>>> >>>> >>>> -- >>>> ---------------------------------------------------------- >>>> >>>> *Nuwan Chamara Pallewela* >>>> >>>> >>>> *Software Engineer* >>>> >>>> *WSO2, Inc. *http://wso2.com >>>> *lean . enterprise . middleware* >>>> >>>> Email *nuw...@wso2.com <nuw...@wso2.com>* >>>> Mobile *+94719079739 <%2B94719079739>@* >>>> >>>> >>>> >>> >>> >>> -- >>> ---------------------------------------------------------- >>> >>> *Nuwan Chamara Pallewela* >>> >>> >>> *Software Engineer* >>> >>> *WSO2, Inc. *http://wso2.com >>> *lean . enterprise . middleware* >>> >>> Email *nuw...@wso2.com <nuw...@wso2.com>* >>> Mobile *+94719079739 <%2B94719079739>@* >>> >>> >>> >>> _______________________________________________ >>> Dev mailing list >>> d...@wso2.org >>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>> >>> >> >> >> -- >> *Imesh Gunaratne* >> Senior Technical Lead >> WSO2 Inc: http://wso2.com >> T: +94 11 214 5345 M: +94 77 374 2057 >> W: http://imesh.gunaratne.org >> Lean . Enterprise . Middleware >> >> > > > -- > Nathasha Naranpanawa > Software Engineering Intern > WSO2 Inc. > > Email: natha...@wso2.com > Mobile: +94775496142 > LinkedIn: https://lk.linkedin.com/in/nathashanaranpanawa > > -- *Imesh Gunaratne* Senior Technical Lead WSO2 Inc: http://wso2.com T: +94 11 214 5345 M: +94 77 374 2057 Lean . Enterprise . Middleware
_______________________________________________ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
