Sorry, readers of this list for not replying, I am very busy
investigating the problem, I try replying later.
Bert
On 16-01-12 23:55, André Warnier wrote:
Hi.
I'll follow-up with an "out of band" message just for now.
It was not very clear from your initial explanations whether you were
sending all of the 1000, 2000, .. 8000 "person's data records" at
once, in one HTTP POST, or sending the data in a person-by-person way.
So my wild guess was based on the symptoms you were providing, which
looked very much like a memory problem caused by the processing of
increasingly large XML documents using a DOM-like parser.
And sorry for missing your references to the platform and Tomcat used.
Jumping ahead however, as Pid noted after your memory figures, it
would appear that on your system, about 1 GB of swap space on disk
/is/ being used, which would be relatively unusual for a Linux system
if this application is the only one that is running.
If indeed this swap space is being used by the Tomcat process, then it
would definitely explain why the processing is slow (but the reason
why that is so would still need to be found).
(As an idea : as soon as an application starts to need the swap area,
you can easily divide its performance by a factor 10).
It would be interesting to see what the memory usage of the
Tomcat-related processes is, while processing these requests. Your
"top" utility would also give an example of this (use CTRL-F, n , <CR>
to sort the processes by memory usage).
The general point is : there is no specific reason, per se, why Tomcat
would slow down dramatically when processing 1300 requests, as
compared to 1000 similar requests. One would expect the 1300 requests
to need approximately 130% of the time needed to process 1000
requests, but not the double amount of time.
So if it is not the back-end database, and it is not the Tomcat code
per se, that leaves
- the web application (including its Axis2 components)
- or some fundamentally different way of handling the data, between
- on one side the client + the stand-alone Axis2 server
- on the other side the client + the Axis2 Web Service webapp
this causing, perhaps, Tomcat to become "saturated" and start trashing
I do not really know Axis2, other than what I just read on the
Wikipedia page.
So I can only make some further guesses at this stage :
You mention that when running Axis2 stand-alone, the processing is
relatively fast (8000 "persons" in 4 seconds), while when running "the
same thing" as a Web service under Tomcat, it is infinitely slower.
I will assume for now that the background database updates are the
same in both cases, and that this is thus not a factor.
You also seem to say that when running Axis2 stand-alone, the 8000
"persons" data is only one POST, while through HTTP it would be 8000
POSTs (or maybe I misunderstood that also).
In any case, can it be that you are comparing apples with oranges ?
As a loose collection of suggestions :
- when you send SOAP requests via HTTP, there is an extra layer of
wrapping around the SOAP data, both on the client side (wrap as
SOAP-HTTP request) and on the server side (unwrap the SOAP-HTTP
wrapping). One would expect some impact of these in any case, if
repeated several thousand times in one case, and not in the other.
- Axis2 seems to allow for configuring entirely different styles of
XML parsing (DOM or stream/event-based). When your client talks to
the Axis2 stand-alone server, is it really sending the data the same
way, and is the Axis2 server library parsing the data the same way ?
Asked another way : is it perhaps so that when you send the data via
HTTP to the Web Service, the Web Service code uses a DOM-like parser,
while when running in stand-alone mode, the Axis2 server uses a
stream-like parsing method ?
- maybe the Axis2 Web Service webapp library leaks memory ?
- maybe your webapp using that library leaks memory ?
- are your client and the server running on the same host ? SOAP is
XML. XML processing, particularly in DOM-mode, can use a significant
amount of memory (*). If the client (to create the XML data) and the
server (to parse it) are running on the same host, and more or less
synchronously, you are doubling the amount of memory required to
process your request/response, and that could explain the onset of
swapping.
- when your client sends requests to the Web Servive over HTTP, does
it send several requests in parallel ? How many Tomcat threads are
available for processing requests, and how many are effectively
processing requests in parallel ?
- maybe indeed Tomcat is spending a lot of its time doing memory
garbage collection, because of one of the above circumstances
(*) an estimate is that to represent an XML document in memory using a
DOM representation, you easily need 10 times the original amount of
data in terms of RAM.
Stream-style parsing and processing is much faster and much leaner in
terms of memory requirements.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
