Hi Amila,
"I think then it is better to keep the httpClient in a local thread
variable rather than in the axis2Configuration."
This will cause some problems with clients who make bunch of calls, but in
different threads. For example a customer may want to reuse the http
connection in his product. Let's suggest that they do thousands of
invocations and every invocation is in a different thread, and they want to
reuse the HttpClient. The problem is that ThreadLocal "independently
initialized copy of the variable". Anyway I think that it is possible to
keep the non-thread-safe part of the HttpClient in a local thread variable.
I.e. HttpState? Then when AbstractHTTPSender#executeMethod is invoked
underneath the ThreadLocal HttpState will be passed as a parameter.
Expecting your comments.
Regards,
Dobri
On Mon, Mar 23, 2009 at 1:08 PM, Amila Suriarachchi <
amilasuriarach...@gmail.com> wrote:
> hi Dobri,
>
> Thank you very much for your response.
>
> On Mon, Mar 23, 2009 at 2:50 PM, Dobri Kitipov <
> kdobrik.ax...@googlemail.com> wrote:
>
>> Hi Amila,
>> I want to share some knowledge about this since these days I am doing some
>> tests with HttpClient reuse.
>> What I want to achieve is to reuse HttpClient among different threads
>> which in turn could invoke different web services (WSs) deployed at
>> different hosts.
>>
>> In fact I think I achieved this requirement, but there are some issues.
>>
>>
>> On Sun, Mar 22, 2009 at 12:52 PM, Amila Suriarachchi <
>> amilasuriarach...@gmail.com> wrote:
>>
>>> hi all,
>>>
>>> recently I came across this link and according to this[1] creating new
>>> http client for every request is not good
>>> in performance wise.
>>>
>>> Current get Http client method looks like this
>>>
>>> protected HttpClient getHttpClient(MessageContext msgContext) {
>>> HttpClient httpClient;
>>> Object reuse =
>>> msgContext.getOptions().getProperty(HTTPConstants.REUSE_HTTP_CLIENT);
>>> if (reuse == null) {
>>> reuse =
>>> msgContext.getConfigurationContext().getProperty(HTTPConstants.REUSE_HTTP_CLIENT);
>>> }
>>> if (reuse != null && JavaUtils.isTrueExplicitly(reuse)) {
>>> httpClient = (HttpClient)
>>> msgContext.getOptions().getProperty(HTTPConstants.CACHED_HTTP_CLIENT);
>>> if (httpClient == null) {
>>> httpClient = (HttpClient)
>>> msgContext.getConfigurationContext()
>>> .getProperty(HTTPConstants.CACHED_HTTP_CLIENT);
>>> }
>>> if (httpClient != null)
>>> return httpClient;
>>> MultiThreadedHttpConnectionManager connectionManager =
>>> new MultiThreadedHttpConnectionManager();
>>> httpClient = new HttpClient(connectionManager);
>>> msgContext.getConfigurationContext()
>>> .setProperty(HTTPConstants.CACHED_HTTP_CLIENT,
>>> httpClient);
>>> } else {
>>> HttpConnectionManager connManager =
>>> (HttpConnectionManager) msgContext.getProperty(
>>>
>>> HTTPConstants.MULTITHREAD_HTTP_CONNECTION_MANAGER);
>>> if (connManager == null) {
>>> connManager =
>>> (HttpConnectionManager) msgContext.getProperty(
>>>
>>> HTTPConstants.MUTTITHREAD_HTTP_CONNECTION_MANAGER);
>>> }
>>> if(connManager != null){
>>> httpClient = new HttpClient(connManager);
>>> } else {
>>> //Multi threaded http connection manager has set as the
>>> default
>>> connManager = new MultiThreadedHttpConnectionManager();
>>> httpClient = new HttpClient(connManager);
>>> }
>>> }
>>>
>>> // Get the timeout values set in the runtime
>>> initializeTimeouts(msgContext, httpClient);
>>> return httpClient;
>>> }
>>>
>>> According to the above article creating many http client instances are
>>> not efficient. But if a user switch on the
>>> HTTPConstants.REUSE_HTTP_CLIENT then he can not give an
>>> MultiThreadedHttpConnectionManager object where users can define maximum
>>> pool size, host configurations etc...
>>>
>>> Therefore I think it is better to have these features,
>>>
>>> 1. Let users to create the MultiThreadedHttpConnectionManager with its
>>> own even when reusing http client.
>>> 2. Does Http Client thread safe? if not we need to keep the cache copy of
>>> the httpClient in a thread local variable.
>>>
>>
>> In short, whereas MultiThreadedHttpConnectionManager used by the
>> HttpClient is thread-safe the HttpClient is claimed to be thread-safe, too
>> [1]. I did not observe any problems during my tests, too.
>>
>> But if different threads are invoking different WSs at different hosts
>> there are some additional issues. The problem is that HttpClient has a
>> HttpState.
>>
>> If you look at HttpState implementation you will see that its intention is
>> to keep any Cookies and credentials. I have no experience of using Cookies
>> with WSs, so I can not comment on this. In fact, I looked at the Axis2 code
>> and it seems that whenever httpClient.executeMethod(...) is invoked with
>> (HostConfiguration config, HttpMethod method) parameters and the HttpState
>> is null (see AbstractHTTPSender#executeMethod). Following the source it
>> seems that for every invocation a new HttpState is created. I am not sure if
>> this is correct or not, but I must confess I did not dive deeply enough into
>> this. Looking at [1] you can read that in this case you need to maintain a
>> separate instance of HttpState per execution thread and pass it as a
>> parameter to the HttpClient#executeMethod.
>>
>
> I think then it is better to keep the httpClient in a local thread
> variable rather than in the axis2Configuration.
>
>>
>>
>>>
>>> Does any one knows how to switch on the keep alive and will it make any
>>> performance improvement?
>>
>>
>> What I know about keepAlive is that keepAlive is used by default when HTTP
>> 1.1 is at hand [2]. Additionally there is a JAVA System property keepAlive
>> that by default has as value "true".
>> Defenetly there is a performance improvment when keepAlive is used.
>>
>
> I did some performance test with the fireAndForget method. If I use this
> without making any configurations then it opens huge number of connections
> since a new connection manager object is created for each invocation. This
> is a serious problem if this code runs inside a service class. (i.e service
> class use this method to call another web service.)
>
> Then I set a MultiThreadedHttpConnectionManager to use it for all
> invocations.
>
> i.e
> client.getOptions().setProperty(HTTPConstants.MULTITHREAD_HTTP_CONNECTION_MANAGER,
> httpConnectionManager);
>
> In this case there was a performance hit.
> AxisEngine.send method creates new thread to send the message. This causes
> a lot of threads and that makes a performance hit to
> MultiThreadedHttpConnectionManager.
>
> The property MessageContext.TRANSPORT_NON_BLOCKING is set at the
> OutOnlyAxisOperationClient.executeImpl method.
>
> When I remove this setting
> i.e.
> if (!block) {
> mc.setProperty(MessageContext.TRANSPORT_NON_BLOCKING,
> Boolean.TRUE);
> }
>
> It worked fine. And I saw all the messages goes in the same tcp channel
> (i.e using keep alive) and actually it is quite fast.
>
> so shall we remove this line?
> IMHO Just waiting for 202 Acepted header won't make any performance issue.
>
> thanks,
> Amila.
>
>
>>
>>
>> [1]
>> http://www.mail-archive.com/httpclient-us...@hc.apache.org/msg01944.html
>> [2]
>> http://www.io.com/~maus/HttpKeepAlive.html<http://www.io.com/%7Emaus/HttpKeepAlive.html>
>>
>> Hope this helps,
>> dobri
>>
>>
>>>
>>>
>>> Anyway I still believe releasing connections should be done at the user
>>> level since it degradates the performance.
>>>
>>> thanks,
>>> Amila.
>>>
>>>
>>> [1] http://hc.apache.org/httpclient-3.x/performance.html
>>>
>>>
>>> On Tue, Mar 10, 2009 at 2:26 AM, Andreas Veithen <
>>> andreas.veit...@gmail.com> wrote:
>>>
>>>> I agree that we also need to consider OperationClient and the
>>>> non-blocking methods. I think that the first step towards a solid
>>>> solution is actually to write a couple of test cases that provide
>>>> evidence for these issues and that we can later use for regression
>>>> testing, but I will have to review the existing unit tests we have.
>>>>
>>>> Andreas
>>>>
>>>> On Wed, Mar 4, 2009 at 01:21, Alexis Midon <mi...@intalio.com> wrote:
>>>> > Another case where "Options#setCallTransportCleanup for
>>>> OperationClient" is
>>>> > obvious is when you call OperationClient#execute in a non-blocking
>>>> way.
>>>> > The caller cannot clean up the transport safely, because the execution
>>>> might
>>>> > still be in progress. In that case it's OperationClient responsability
>>>> to
>>>> > clean up the transport.
>>>> >
>>>> > Alexis
>>>> >
>>>> >
>>>> > On Mon, Mar 2, 2009 at 10:11 AM, Alexis Midon <mi...@intalio.com>
>>>> wrote:
>>>> >>
>>>> >> There is still some inconsistencies between how
>>>> ServiceClient#sendReceive
>>>> >> and Operation#execute use Options#getCallTransportCleanup.
>>>> >>
>>>> >> And I think that would help a lot if the related jira issues get
>>>> cleaned
>>>> >> up a little bit.
>>>> >>
>>>> >> Thanks for your time and feedback.
>>>> >>
>>>> >> Alexis
>>>> >>
>>>> >>
>>>> >> On Sun, Mar 1, 2009 at 9:02 PM, Amila Suriarachchi
>>>> >> <amilasuriarach...@gmail.com> wrote:
>>>> >>>
>>>> >>> hi all,
>>>> >>>
>>>> >>> On Fri, Feb 27, 2009 at 6:35 PM, Andreas Veithen
>>>> >>> <andreas.veit...@gmail.com> wrote:
>>>> >>>>
>>>> >>>> I think that callTransportCleanup should never be turned on by
>>>> default
>>>> >>>> because it would disable deferred parsing of the response. What
>>>> needs
>>>> >>>> to be done urgently is to improve the documentation of the
>>>> >>>> ServiceClient class to make it clear that it is mandatory to either
>>>> >>>> call cleanupTransport explicitly or to set callTransportCleanup to
>>>> >>>> true. Also the cleanupTransport method itself doesn't have any
>>>> >>>> Javadoc.
>>>> >>>
>>>> >>> thanks for in-depth analysing of this issue. If the issue is not
>>>> calling
>>>> >>> to transport clean up then I clearly agree with what Andreas.
>>>> >>>
>>>> >>> Axis2 uses deffered building. There when the user gets the response
>>>> >>> OMElement it has not build and Axis2 does not know when he going to
>>>> finish
>>>> >>> with it. So it is responsibility of the user to call the transport
>>>> clean up
>>>> >>> once they have done with the OMElement.
>>>> >>>
>>>> >>> But this problem does not occur with the generated data bind code.
>>>> There
>>>> >>> before returning from the stub method it generates the all the
>>>> databinding
>>>> >>> classes. In the generated code finally method calls
>>>> >>>
>>>> _messageContext.getTransportOut().getSender().cleanup(_messageContext);
>>>> >>>
>>>> >>> to clean up the transport.
>>>> >>>
>>>> >>> thanks,
>>>> >>> Amila.
>>>> >>>>
>>>> >>>> Andreas
>>>> >>>>
>>>> >>>> On Fri, Feb 27, 2009 at 12:19, Dobri Kitipov
>>>> >>>> <kdobrik.ax...@googlemail.com> wrote:
>>>> >>>> > Hi Andreas,
>>>> >>>> > thank you for the comment. I think you get the question.
>>>> >>>> > Quick test shows that setting the following line of code into the
>>>> >>>> > client:
>>>> >>>> >
>>>> >>>> > options.setCallTransportCleanup(true);
>>>> >>>> >
>>>> >>>> > forces the closure of the http connection. It seems it is not the
>>>> >>>> > default
>>>> >>>> > behavior. This is a good and fast solution. I was a little bit
>>>> more
>>>> >>>> > focused
>>>> >>>> > on wondering why I have such a difference b/n using SOAP and MIME
>>>> >>>> > builder.
>>>> >>>> >
>>>> >>>> > I need to think about some use cases when we need to have
>>>> >>>> > options.setCallTransportCleanup(false). Can we have this by
>>>> default in
>>>> >>>> > some
>>>> >>>> > cases?
>>>> >>>> >
>>>> >>>> > Anyway, it will be worth having a further analysis of the issue
>>>> we
>>>> >>>> > have with
>>>> >>>> > SOAPBuilder behavior.
>>>> >>>> >
>>>> >>>> > Thank you,
>>>> >>>> > Dobri
>>>> >>>> >
>>>> >>>> >
>>>> >>>> > On Fri, Feb 27, 2009 at 12:46 PM, Andreas Veithen
>>>> >>>> > <andreas.veit...@gmail.com> wrote:
>>>> >>>> >>
>>>> >>>> >> If I understand correctly, Dobri's findings can be summarized as
>>>> >>>> >> follows:
>>>> >>>> >> 1. Once the InputStream is consumed, commons-httpclient
>>>> automatically
>>>> >>>> >> releases the connection.
>>>> >>>> >> 2. SOAPBuilder never completely consumes the InputStream.
>>>> >>>> >>
>>>> >>>> >> The SOAPBuilder behavior is indeed somewhat questionable, but it
>>>> is
>>>> >>>> >> important to understand that because of the deferred parsing
>>>> model
>>>> >>>> >> used by Axiom, there is never a guarantee that the InputStream
>>>> will
>>>> >>>> >> be
>>>> >>>> >> completely consumed. Normally releasing the connection is the
>>>> >>>> >> responsibility of the CommonsHTTPTransportSender#cleanup method
>>>> which
>>>> >>>> >> should be called by ServiceClient#cleanupTransport. It would be
>>>> >>>> >> interesting to know if that method is called, and if it is, why
>>>> it
>>>> >>>> >> fails to release the connection.
>>>> >>>> >>
>>>> >>>> >> Andreas
>>>> >>>> >>
>>>> >>>> >> On Fri, Feb 27, 2009 at 10:10, Dobri Kitipov
>>>> >>>> >> <kdobrik.ax...@googlemail.com> wrote:
>>>> >>>> >> > Hi all,
>>>> >>>> >> > I have observed absolutely the same thing these days. I need
>>>> some
>>>> >>>> >> > more
>>>> >>>> >> > time
>>>> >>>> >> > to analyze the whole picture, but here is my current synthesis
>>>> of
>>>> >>>> >> > the
>>>> >>>> >> > issue.
>>>> >>>> >> >
>>>> >>>> >> > It seems that http connection release is tightly coupled with
>>>> the
>>>> >>>> >> > Axis2
>>>> >>>> >> > builder used to handle and process the response body and the
>>>> >>>> >> > corresponding
>>>> >>>> >> > input streams used. The builder and the InputStream used are
>>>> based
>>>> >>>> >> > on
>>>> >>>> >> > the
>>>> >>>> >> > HTTP headers fields like "Content-Type" and
>>>> "Transfer-Encoding"
>>>> >>>> >> > (e.g.
>>>> >>>> >> > Content-Type: text/xml; charset=UTF-8 Transfer-Encoding:
>>>> chunked).
>>>> >>>> >> > So
>>>> >>>> >> > if we
>>>> >>>> >> > have Content-Type: text/xml; then SOAPBuilder class will be
>>>> used.
>>>> >>>> >> > If we
>>>> >>>> >> > have type="application/xop+xml" then MIMEBuilder will be
>>>> used.
>>>> >>>> >> >
>>>> >>>> >> > The successfull story when we have MIMIBuilder:
>>>> >>>> >> >
>>>> >>>> >> > When MIMEBuilder is used then the response Buffered
>>>> InputStream
>>>> >>>> >> > (IS) is
>>>> >>>> >> > wrrapped (I will use "->" sign as substitution for wrrapped)
>>>> >>>> >> > ChunkedIS
>>>> >>>> >> > ->
>>>> >>>> >> > AutoCloseIS (this has a responseConsumed() method notified
>>>> when
>>>> >>>> >> > IS.read()
>>>> >>>> >> > returns -1, which means that the response IS has been
>>>> completely
>>>> >>>> >> > read)
>>>> >>>> >> > ->
>>>> >>>> >> > PushBackIS ->BounderyPushBackIS. The BounderyPushBackIS reads
>>>> the
>>>> >>>> >> > response
>>>> >>>> >> > stream (see readFromStream(....)) in a cycle till it reaches
>>>> its
>>>> >>>> >> > end. At
>>>> >>>> >> > every iteration of this cycle a AutoCloseIS checkClose(l) is
>>>> >>>> >> > invoked. So
>>>> >>>> >> > when the end is reached (-1 is returned) then this check
>>>> causes the
>>>> >>>> >> > invokation of the AutoCloseIS checkClose(...) method. This
>>>> method
>>>> >>>> >> > invokes
>>>> >>>> >> > notifyWatcher() that in turn invokes responseBodyConsumed()
>>>> method
>>>> >>>> >> > of
>>>> >>>> >> > the
>>>> >>>> >> > HttpMethodBase class. This causes the release of the http
>>>> >>>> >> > connection
>>>> >>>> >> > which
>>>> >>>> >> > is returned back to the connection pool. So here we have no
>>>> problem
>>>> >>>> >> > with
>>>> >>>> >> > connection reuse.
>>>> >>>> >> >
>>>> >>>> >> > The bad story we have with SOAPBuilder:
>>>> >>>> >> >
>>>> >>>> >> > When SOAPBuilder is used then the response Buffered
>>>> InputStream is
>>>> >>>> >> > wrrapped
>>>> >>>> >> > in a ChunkedIS -> AutoCloseIS -> PushBackIS. May be you has
>>>> noticed
>>>> >>>> >> > that
>>>> >>>> >> > BounderyPushBackIS is not used. As a result the respose IS is
>>>> not
>>>> >>>> >> > completely
>>>> >>>> >> > read (in fact this is not really correct, it could be read but
>>>> the
>>>> >>>> >> > invokation of the PushBackIS unread(...) method causes the
>>>> >>>> >> > AutoCloseIS
>>>> >>>> >> > checkClose() method never to return -1). As a result the http
>>>> >>>> >> > connection
>>>> >>>> >> > is
>>>> >>>> >> > not released. And since there is a limit to have 2 connection
>>>> per
>>>> >>>> >> > host
>>>> >>>> >> > then after the second invokation of the WS client the thread
>>>> hangs
>>>> >>>> >> > waiting
>>>> >>>> >> > for a free connection.
>>>> >>>> >> >
>>>> >>>> >> > Please, provide us with your comments on this issue.
>>>> >>>> >> >
>>>> >>>> >> > Thank you in advance.
>>>> >>>> >> >
>>>> >>>> >> > Regards,
>>>> >>>> >> > Dobri
>>>> >>>> >> >
>>>> >>>> >> > On Fri, Feb 27, 2009 at 3:54 AM, Alexis Midon <
>>>> mi...@intalio.com>
>>>> >>>> >> > wrote:
>>>> >>>> >> >>
>>>> >>>> >> >> no taker for an easy patch?
>>>> >>>> >> >>
>>>> >>>> >> >> Alexis
>>>> >>>> >> >>
>>>> >>>> >> >>
>>>> >>>> >> >> On Wed, Feb 25, 2009 at 6:45 PM, Alexis Midon <
>>>> mi...@intalio.com>
>>>> >>>> >> >> wrote:
>>>> >>>> >> >>>
>>>> >>>> >> >>> Hi everyone,
>>>> >>>> >> >>>
>>>> >>>> >> >>> All the issues relatives to AXIS2-935 are really messy, some
>>>> of
>>>> >>>> >> >>> them
>>>> >>>> >> >>> are
>>>> >>>> >> >>> closed but their clones are not. Some are flagged as fixed
>>>> but
>>>> >>>> >> >>> are
>>>> >>>> >> >>> obviously
>>>> >>>> >> >>> not. All these issues are really old, so I'd like to take a
>>>> >>>> >> >>> chance to
>>>> >>>> >> >>> bring
>>>> >>>> >> >>> them back to your attention, especially before releasing
>>>> 1.5.
>>>> >>>> >> >>>
>>>> >>>> >> >>> I'll post a description of the issue in this email as a
>>>> summary
>>>> >>>> >> >>> all
>>>> >>>> >> >>> the
>>>> >>>> >> >>> jiras.
>>>> >>>> >> >>>
>>>> >>>> >> >>> By default, ServiceClient uses one HttpConnectionManager per
>>>> >>>> >> >>> invocation
>>>> >>>> >> >>> [2]. This connection manager will create and provide one
>>>> >>>> >> >>> connection to
>>>> >>>> >> >>> HTTPSender. The first issue is that by default this
>>>> connection is
>>>> >>>> >> >>> never
>>>> >>>> >> >>> released to the pool [3]. if you do zillions of invocations,
>>>> this
>>>> >>>> >> >>> leak
>>>> >>>> >> >>> will
>>>> >>>> >> >>> max out your number of file descriptors.
>>>> >>>> >> >>>
>>>> >>>> >> >>> Your investigations in Axis2 options quickly lead you to the
>>>> >>>> >> >>> REUSE_HTTP_CLIENT option. But this first issue has some
>>>> >>>> >> >>> unfortunate
>>>> >>>> >> >>> consequences if you activate it. Actually if you do so, a
>>>> single
>>>> >>>> >> >>> connection
>>>> >>>> >> >>> manager is shared across all invocations. But because
>>>> connections
>>>> >>>> >> >>> are
>>>> >>>> >> >>> not
>>>> >>>> >> >>> release, the pool is starved after two invocations, and the
>>>> third
>>>> >>>> >> >>> invocation
>>>> >>>> >> >>> hangs out indefinitely. :(
>>>> >>>> >> >>>
>>>> >>>> >> >>> If you keep digging you will find the
>>>> AUTO_RELEASE_CONNECTION
>>>> >>>> >> >>> option.
>>>> >>>> >> >>> Its
>>>> >>>> >> >>> sounds like a good lead! Let's try it. If you activate this
>>>> >>>> >> >>> option the
>>>> >>>> >> >>> connection is properly released -Yahoooo! the leak is fixed
>>>> - but
>>>> >>>> >> >>> unfortunately a new issue shows up (issue #2, aka
>>>> AXIS2-3478).
>>>> >>>> >> >>> AbstractHTTPSender passes the stream of the connection to
>>>> the
>>>> >>>> >> >>> message
>>>> >>>> >> >>> context [4] , but that the connection is now properly
>>>> released,
>>>> >>>> >> >>> so
>>>> >>>> >> >>> this
>>>> >>>> >> >>> stream is closed before the SOAPBuilder gets a chance to
>>>> read the
>>>> >>>> >> >>> response
>>>> >>>> >> >>> body.
>>>> >>>> >> >>> Boom! "IOException: Attempted read on closed stream"
>>>> >>>> >> >>>
>>>> >>>> >> >>> These issues are easily reproducible in versions 1.3, 1.4,
>>>> 1.5.
>>>> >>>> >> >>>
>>>> >>>> >> >>> I submitted and documented a fix in AXIS2-2931 [5], if you
>>>> had a
>>>> >>>> >> >>> chance
>>>> >>>> >> >>> to look at it that would be much appreciate.
>>>> >>>> >> >>>
>>>> >>>> >> >>>
>>>> >>>> >> >>> Alexis
>>>> >>>> >> >>>
>>>> >>>> >> >>>
>>>> >>>> >> >>> [1]
>>>> >>>> >> >>>
>>>> >>>> >> >>>
>>>> >>>> >> >>>
>>>> https://issues.apache.org/jira/browse/AXIS2-935?focusedCommentId=12513543#action_12513543
>>>> >>>> >> >>> [2] see method getHttpClient in
>>>> >>>> >> >>>
>>>> >>>> >> >>>
>>>> >>>> >> >>>
>>>> https://svn.apache.org/repos/asf/webservices/commons/trunk/modules/transport/modules/http/src/org/apache/axis2/transport/http/AbstractHTTPSender.java
>>>> >>>> >> >>> [3] see method cleanup in
>>>> >>>> >> >>>
>>>> >>>> >> >>>
>>>> >>>> >> >>>
>>>> https://svn.apache.org/repos/asf/webservices/commons/trunk/modules/transport/modules/http/src/org/apache/axis2/transport/http/HTTPSender.java
>>>> >>>> >> >>> [4] see method processResponse in AbstractHTTPSender.java
>>>> >>>> >> >>> [5]
>>>> >>>> >> >>>
>>>> >>>> >> >>>
>>>> >>>> >> >>>
>>>> https://issues.apache.org/jira/browse/AXIS2-2931?focusedCommentId=12676837#action_12676837
>>>> >>>> >> >>
>>>> >>>> >> >
>>>> >>>> >> >
>>>> >>>> >
>>>> >>>> >
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> --
>>>> >>> Amila Suriarachchi
>>>> >>> WSO2 Inc.
>>>> >>> blog: http://amilachinthaka.blogspot.com/
>>>> >>
>>>> >
>>>> >
>>>>
>>>
>>>
>>>
>>> --
>>> Amila Suriarachchi
>>> WSO2 Inc.
>>> blog: http://amilachinthaka.blogspot.com/
>>>
>>
>>
>
>
> --
> Amila Suriarachchi
> WSO2 Inc.
> blog: http://amilachinthaka.blogspot.com/
>
