Can we profile and check why there is a drop in performance in high concurrency?
On Wed, Mar 2, 2016 at 2:51 PM, Nathasha Naranpanawa <natha...@wso2.com> wrote: > Hi, > > As initially planned, the feature was implemented to collect web > application statistics and publish them to DAS. > > Given below is an illustration of the high-level architecture of the > feature; > > > Currently, all configuration parameters required to create a connection to > DAS (such as DAS username, password, urls, etc.) are provided as valve > attributes in the server.xml file. The events are published through a > Thrift data agent and publishing is handled by the DataPublisher API which > is asynchronous. > > If DAS is not reachable, all the events will be buffered in order. When > DAS is made available, the buffered events will be pushed to DAS and > published. This is handled by the DataPublisher API itself. > > A performance testing was also carried out using Jmeter in order to > compare the throughput with and without the publisher valve. Following is > an indication of the throughput of the server according to a set of data > obtained; > > > > The Spark script which is currently in the Capp is a sample one. The Cron > expression in the script is scheduled to execute the script at every > second. The script presents two tables, raw statistics data and request > summary per minute. The table that holds raw statistics data are updated at > every invoke of a web application. The request summary per minute table is > only updated at each minute of invoke. Summarizing information published to > DAS and displaying them to users is done as a separate project by another > intern by creating gadgets to be deployed on the Dashboard server. > > Any further suggestions and feedback are highly appreciated. > > Thanks, > Nathasha. > > > On Tue, Dec 15, 2015 at 5:04 PM, Imesh Gunaratne <im...@wso2.com> wrote: > >> >> On Tue, Dec 15, 2015 at 11:29 AM, Nathasha Naranpanawa <natha...@wso2.com >>>> > wrote: >>>> >>>>> >>>>> - Adding a custom Valve to the tomcat server in order to collect >>>>> data from the server. >>>>> >>>>> Can you please explain more details about this valve? >> >> - What would be the approach taken to publish data in asynchronous >> mode? >> - How do we plan to share data between the thread that accept the >> request and the thread that publish statistics? >> - How would that scale/affect if the incoming requests grow? >> - How would this valve would handle if DAS is not reachable? >> >> Thanks >> >>> >>>>> - Creating necessary CApp packages in order to define means of >>>>> publishing data from tomcat server to DAS, receiving and storing the >>>>> data. >>>>> >>>>> The custom tomcat Valve behaves as an Event Publisher and publishes >>>>> data to DAS through Thrift data agent. For each request from the server, >>>>> an >>>>> Event is created including metadata and payload data and is published to >>>>> the DAS by a DataPublisher. >>>>> >>>>> A CApp was created by including the definitions for an Event Stream, >>>>> Event Receiver and Event Store to receive data from the publisher valve >>>>> and >>>>> store them. This is deployed in the >>>>> wso2das-3.0.0/repository/deployment/server/carbonapps folder. >>>>> >>>>> A Spark script is also added to the CApp in order to query the data >>>>> received by DAS. The script presents a summary of the requests received by >>>>> DAS per minute. The summary includes the following information; >>>>> >>>>> - average request count >>>>> - average response time >>>>> - HTTP request success count >>>>> - HTTP request error count >>>>> >>>>> Any suggestions and feedback are highly appreciated. >>>>> >>>>> Thanks, >>>>> >>>>> >>>>> -- >>>>> Nathasha Naranpanawa >>>>> Software Engineering Intern >>>>> WSO2 Inc. >>>>> >>>>> Email: natha...@wso2.com >>>>> Mobile: +94775496142 >>>>> LinkedIn: https://lk.linkedin.com/in/nathashanaranpanawa >>>>> >>>>> >>>>> _______________________________________________ >>>>> Dev mailing list >>>>> d...@wso2.org >>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>> >>>>> >>>> >>>> >>>> -- >>>> ---------------------------------------------------------- >>>> >>>> *Nuwan Chamara Pallewela* >>>> >>>> >>>> *Software Engineer* >>>> >>>> *WSO2, Inc. *http://wso2.com >>>> *lean . enterprise . middleware* >>>> >>>> Email *nuw...@wso2.com <nuw...@wso2.com>* >>>> Mobile *+94719079739 <%2B94719079739>@* >>>> >>>> >>>> >>> >>> >>> -- >>> ---------------------------------------------------------- >>> >>> *Nuwan Chamara Pallewela* >>> >>> >>> *Software Engineer* >>> >>> *WSO2, Inc. *http://wso2.com >>> *lean . enterprise . middleware* >>> >>> Email *nuw...@wso2.com <nuw...@wso2.com>* >>> Mobile *+94719079739 <%2B94719079739>@* >>> >>> >>> >>> _______________________________________________ >>> Dev mailing list >>> d...@wso2.org >>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>> >>> >> >> >> -- >> *Imesh Gunaratne* >> Senior Technical Lead >> WSO2 Inc: http://wso2.com >> T: +94 11 214 5345 M: +94 77 374 2057 >> W: http://imesh.gunaratne.org >> Lean . Enterprise . Middleware >> >> > > > -- > Nathasha Naranpanawa > Software Engineering Intern > WSO2 Inc. > > Email: natha...@wso2.com > Mobile: +94775496142 > LinkedIn: https://lk.linkedin.com/in/nathashanaranpanawa > > > _______________________________________________ > Architecture mailing list > Architecture@wso2.org > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- *Kishanthan Thangarajah* Associate Technical Lead, Platform Technologies Team, WSO2, Inc. lean.enterprise.middleware Mobile - +94773426635 Blog - *http://kishanthan.wordpress.com <http://kishanthan.wordpress.com>* Twitter - *http://twitter.com/kishanthan <http://twitter.com/kishanthan>*
_______________________________________________ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
