There are two sides to this problem. One is that, if it is failing for
some scenario, we should be able to re-produce it. So without some code
to re-produce the fault, we cannot come to a conclusion that it fails.
If accepts fails, and it crashes, then there needs to be someuser
scenario that makes this happen. If we cannot find a user scenario that
makes accept to fail, then even though the bug is there, it will never
be exposed.
Anyway we can ask the user to try with your fixes for accept failure
case and test to see if the problem persists.
Samisa...
Senaka Fernando wrote:
Hi again Samisa,
And, also, I don't have any similar database accessing logic etc.
Therefore, it is hard to conclude without seeing the actual code. Also,
the user seems to be using a modified version of "http_server_main.c", it
is actually a cpp file if you go through that portion which I highlighted
in his log. May be some issues due to not handling concurrency properly
could also lead him into trouble.
However, what I can say is that there was a bug if accept(2) was to fail,
according to the tests I did. And, according to literature available
online that can fail. Plus, we do have a mechanism of retrying, if it does
fail which did not work due to the segfault.
Regards,
Senaka
Were you able to re-create the problem without passing -1 to accept?
No I was not able to make accept fail without passing -1. Added a printf
there to print any failure in accept and it didn't. However, ran into the
segfault issue when accept actually failed, which I had to do manually.
I did test with 1000 back-2-back mtom requests in 5 terminals, and also
1000 repeated calls to an echo client that sends 1000 back-2-back
requests, and in 4 terminals.
None of these cases failed. There were occasional hangs but still it
didn't result in any failure on either client or server side.
However, I didn't have any way to test multiple requests made to a single
port from multiple machines. That may be a different scenario.
Regards,
Senaka
Samisa...
Senaka Fernando wrote:
Senaka Fernando wrote:
Hi Samisa,
I fixed two bugs regarding stream handling inside
simple_http_svr_conn.c.
This fixes an issue of this nature.
Say, we have two server threads that try to serve requests offered to
the
axis2 server. Now, according to what is intended, one must serve and
the
other must wait until it can serve. The previous implementation
caused
this to fail with a segfault. But, with the fix, it will rather not
crash
but, loop until it gets a chance to serve the request.
Did you test before the fix and did it seg fault for you? Can you
please
explain the scenario under which it seg faulted? If it is a problem, I
too should be able to re-create it.
And how did you solve the problem? By adding a sleep?
I made the accept(2) call fail. By passing a -1 to it. Once that
failed,
there were two places that segfaulted which I already fixed, few
minutes
ago. No, I didn't add a sleep(). I simply prevented the segfault by
adding
necessary is_null checks before calling methods. Then, the server just
kept on spawning threads and freeing them as it can't serve the
request.
This is the expected behaviour in http_svr_thread.c I believe.
In a real world scenario, a situation similar to a -1 passed to
accept(2)
will be temporal. So a respawned thread down the line should server the
request without any problem.
Passing -1 to accept was done by,
Index: util/src/network_handler.c
===================================================================
--- util/src/network_handler.c (revision 634578)
+++ util/src/network_handler.c (working copy)
@@ -221,7 +221,7 @@
AXIS2_ENV_CHECK(env, AXIS2_CRITICAL_FAILURE);
cli_len = sizeof(cli_addr);
- cli_socket = accept(svr_socket, (struct sockaddr *) &cli_addr,
&cli_len);
+ cli_socket = accept(-1, (struct sockaddr *) &cli_addr, &cli_len);
if (cli_socket < 0)
AXIS2_LOG_ERROR(env->log, AXIS2_LOG_SI,
"[Axis2][network_handler] Socket accept \
---
I'm not sure whether this is what really lead to the core dump, or was
it
something else.
This was all that I could try on based on the less informative
valgrind.log that was sent. :(...
Also i did test 1000 back-to-back requests and they all were served
without any issues.
Regards,
Senaka
Samisa...
Could this solve the core dump issue?
Regards,
Senaka
Raghavendra SM wrote:
As expected, when I put a delay of about 50 milliseconds between
the
requests there is no dump.
But we would still like to know the following,
+ Isn't Axis capable of handling such huge bombardment of requests
(of
the order of 1000's) one after another without any delay between
the
requests? Has anyone tried such client before?
Yes it is supposed to. We have done performance testing that proved
that
Axis2/C can serve more than 300 rps.
+ Can we make out anything out of the valgrind report or the axis
logs
that were attached?
Unfortunately not. When valgrind runs, it makes the system slow,
that
prevents the system crashing in situations like that you are
experiencing. To solve this kind of a problem, it is a must that you
provide us with some sample code that can be used to reproduce the
problem. Without having a way to re-produce the situation, there is
no
way to fix this problem - it could well be a problem in Axis2/C as
well
as it could also be in the code that you use.
Samisa...
Regards,
~raghav
-----Original Message-----
From: Senaka Fernando [mailto:[EMAIL PROTECTED]
Sent: Thursday, March 06, 2008 4:22 PM
To: Apache AXIS C User List
Subject: RE: Core dumps in Axis libraries
Hi Raghav,
Can you please try this with --enable-guthilla=yes, and send the
dump.log,
and also the appropriate log_files generated by the server/client.
There also can be issues with your "condition variable" being used
in
multiple threads.
It also seems that if you pause between your requests you might get
this
solved. According to what you say, when you increase the log_level,
obviously you waste some time there which might be the solution
rather.
You can use the AXIS2_SLEEP() and AXIS2_USLEEP() inside Axis2/C to
pause
between requests. You can get an idea on how to use these functions
by
taking a look at the Axis2/C user_guide/client samples.
Regards,
Senaka
Thanks,
As suggested, I tried using axis2c-1.3.0 both with and without
--enable-guthilla=yes. But I still see different dumps
consistently.
Please find the attached file with back-traces.
Are you using the same service client instance from multiple
threads?
We have a axis thread pool with default number of threads and a
set
of our own threads(about 10, lets call them "database threads").
+ On receiving a HTTP soap request, axis thread invokes the
appropriate
function that is bound to our logic.
+ Now, the axis thread requests a "database thread" to perform the
appropriate query and waits on a condition variable.
+ our "db thread" performs the necessary query and signals on that
condition variable so that axis thread shall take the control
over.
+ axis thread will create the om output and dispatch as usual.
My doubts:
+ does this design has any apparent drawback when there is a
flurry
of
back-to-back soap requests?
+ Is there a need to increase the axis thread pool size? If yes,
how
to
do it?
+ As told before, the dump is seen during a particular scenario
only.
i.e: while trying to add a list of already existing users to my
database. For each user there is a SOAP request generated and the
requests are sequential, i.e the second request is sent only after
the
response for previous request is received.
+ if I increase the log level of my own module that uses axis2c,
there
is no dump. On increasing the log level, my module writes quite a
bit
of
information to my log file.
Your help is appreciated.
Thanks & Regards,
~raghav
-----Original Message-----
From: Samisa Abeysinghe [mailto:[EMAIL PROTECTED]
Sent: Wednesday, March 05, 2008 7:25 PM
To: Apache AXIS C User List
Subject: Re: Core dumps in Axis libraries
Raghavendra SM wrote:
Thanks Manjula,
But thats not a viable option for us. We would still want to
continue
using axis2C-1.0 for some more time.
Are these dumps well known?
No.
Are you using the same service client instance from multiple
threads?
Samisa...
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
--
Samisa Abeysinghe
Software Architect; WSO2 Inc.
http://www.wso2.com/ - "Oxygenating the Web Service Platform."
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
--
Samisa Abeysinghe
Software Architect; WSO2 Inc.
http://www.wso2.com/ - "Oxygenating the Web Service Platform."
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
--
Samisa Abeysinghe
Software Architect; WSO2 Inc.
http://www.wso2.com/ - "Oxygenating the Web Service Platform."
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
--
Samisa Abeysinghe
Software Architect; WSO2 Inc.
http://www.wso2.com/ - "Oxygenating the Web Service Platform."
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
