> There are two sides to this problem. One is that, if it is failing for > some scenario, we should be able to re-produce it. So without some code > to re-produce the fault, we cannot come to a conclusion that it fails. > If accepts fails, and it crashes, then there needs to be someuser > scenario that makes this happen. If we cannot find a user scenario that > makes accept to fail, then even though the bug is there, it will never > be exposed. > > Anyway we can ask the user to try with your fixes for accept failure > case and test to see if the problem persists.
Yes, that can be done. Actually as I said before, this is just a wild guess that an issue in accept(2) could've created a segfault. Reason is that there is no way to figure out what could be the reason from the valgrind log. Of course if this is a SIGTERM inside accept, he will have to use a signal handler, there was an example in the Fast CGI Issue fix that I sent, and return the appropriate error to the rest of the source following accept(2). Then our code will pick it up correctly from there onwards with this fix in place. Also, adding a signal handler is a user-specific thing which we need not attend to because, AFAIK, Axis2/C implementation will not cause a SIGTERM inside accept(2) and that should be through the intervention of some external source. Regards, Senaka > > Samisa... > > Senaka Fernando wrote: >> Hi again Samisa, >> >> And, also, I don't have any similar database accessing logic etc. >> Therefore, it is hard to conclude without seeing the actual code. Also, >> the user seems to be using a modified version of "http_server_main.c", >> it >> is actually a cpp file if you go through that portion which I >> highlighted >> in his log. May be some issues due to not handling concurrency properly >> could also lead him into trouble. >> >> However, what I can say is that there was a bug if accept(2) was to >> fail, >> according to the tests I did. And, according to literature available >> online that can fail. Plus, we do have a mechanism of retrying, if it >> does >> fail which did not work due to the segfault. >> >> Regards, >> Senaka >> >> >>>> Were you able to re-create the problem without passing -1 to accept? >>>> >>> No I was not able to make accept fail without passing -1. Added a >>> printf >>> there to print any failure in accept and it didn't. However, ran into >>> the >>> segfault issue when accept actually failed, which I had to do manually. >>> >>> I did test with 1000 back-2-back mtom requests in 5 terminals, and also >>> 1000 repeated calls to an echo client that sends 1000 back-2-back >>> requests, and in 4 terminals. >>> >>> None of these cases failed. There were occasional hangs but still it >>> didn't result in any failure on either client or server side. >>> >>> However, I didn't have any way to test multiple requests made to a >>> single >>> port from multiple machines. That may be a different scenario. >>> >>> Regards, >>> Senaka >>> >>> >>>> Samisa... >>>> >>>> Senaka Fernando wrote: >>>> >>>>>> Senaka Fernando wrote: >>>>>> >>>>>> >>>>>>> Hi Samisa, >>>>>>> >>>>>>> I fixed two bugs regarding stream handling inside >>>>>>> simple_http_svr_conn.c. >>>>>>> This fixes an issue of this nature. >>>>>>> >>>>>>> Say, we have two server threads that try to serve requests offered >>>>>>> to >>>>>>> the >>>>>>> axis2 server. Now, according to what is intended, one must serve >>>>>>> and >>>>>>> the >>>>>>> other must wait until it can serve. The previous implementation >>>>>>> caused >>>>>>> this to fail with a segfault. But, with the fix, it will rather not >>>>>>> crash >>>>>>> but, loop until it gets a chance to serve the request. >>>>>>> >>>>>>> >>>>>>> >>>>>> Did you test before the fix and did it seg fault for you? Can you >>>>>> please >>>>>> explain the scenario under which it seg faulted? If it is a problem, >>>>>> I >>>>>> too should be able to re-create it. >>>>>> >>>>>> And how did you solve the problem? By adding a sleep? >>>>>> >>>>>> >>>>> I made the accept(2) call fail. By passing a -1 to it. Once that >>>>> failed, >>>>> there were two places that segfaulted which I already fixed, few >>>>> minutes >>>>> ago. No, I didn't add a sleep(). I simply prevented the segfault by >>>>> adding >>>>> necessary is_null checks before calling methods. Then, the server >>>>> just >>>>> kept on spawning threads and freeing them as it can't serve the >>>>> request. >>>>> This is the expected behaviour in http_svr_thread.c I believe. >>>>> >>>>> In a real world scenario, a situation similar to a -1 passed to >>>>> accept(2) >>>>> will be temporal. So a respawned thread down the line should server >>>>> the >>>>> request without any problem. >>>>> >>>>> Passing -1 to accept was done by, >>>>> >>>>> Index: util/src/network_handler.c >>>>> =================================================================== >>>>> --- util/src/network_handler.c (revision 634578) >>>>> +++ util/src/network_handler.c (working copy) >>>>> @@ -221,7 +221,7 @@ >>>>> AXIS2_ENV_CHECK(env, AXIS2_CRITICAL_FAILURE); >>>>> >>>>> cli_len = sizeof(cli_addr); >>>>> - cli_socket = accept(svr_socket, (struct sockaddr *) &cli_addr, >>>>> &cli_len); >>>>> + cli_socket = accept(-1, (struct sockaddr *) &cli_addr, >>>>> &cli_len); >>>>> if (cli_socket < 0) >>>>> AXIS2_LOG_ERROR(env->log, AXIS2_LOG_SI, >>>>> "[Axis2][network_handler] Socket accept \ >>>>> --- >>>>> >>>>> I'm not sure whether this is what really lead to the core dump, or >>>>> was >>>>> it >>>>> something else. >>>>> >>>>> This was all that I could try on based on the less informative >>>>> valgrind.log that was sent. :(... >>>>> >>>>> Also i did test 1000 back-to-back requests and they all were served >>>>> without any issues. >>>>> >>>>> Regards, >>>>> Senaka >>>>> >>>>> >>>>> >>>>>> Samisa... >>>>>> >>>>>> >>>>>> >>>>>>> Could this solve the core dump issue? >>>>>>> >>>>>>> Regards, >>>>>>> Senaka >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> Raghavendra SM wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> As expected, when I put a delay of about 50 milliseconds between >>>>>>>>> the >>>>>>>>> requests there is no dump. >>>>>>>>> >>>>>>>>> But we would still like to know the following, >>>>>>>>> + Isn't Axis capable of handling such huge bombardment of >>>>>>>>> requests >>>>>>>>> (of >>>>>>>>> the order of 1000's) one after another without any delay between >>>>>>>>> the >>>>>>>>> requests? Has anyone tried such client before? >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> Yes it is supposed to. We have done performance testing that >>>>>>>> proved >>>>>>>> that >>>>>>>> Axis2/C can serve more than 300 rps. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> + Can we make out anything out of the valgrind report or the axis >>>>>>>>> logs >>>>>>>>> that were attached? >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> Unfortunately not. When valgrind runs, it makes the system slow, >>>>>>>> that >>>>>>>> prevents the system crashing in situations like that you are >>>>>>>> experiencing. To solve this kind of a problem, it is a must that >>>>>>>> you >>>>>>>> provide us with some sample code that can be used to reproduce the >>>>>>>> problem. Without having a way to re-produce the situation, there >>>>>>>> is >>>>>>>> no >>>>>>>> way to fix this problem - it could well be a problem in Axis2/C as >>>>>>>> well >>>>>>>> as it could also be in the code that you use. >>>>>>>> >>>>>>>> Samisa... >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> Regards, >>>>>>>>> ~raghav >>>>>>>>> >>>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: Senaka Fernando [mailto:[EMAIL PROTECTED] >>>>>>>>> Sent: Thursday, March 06, 2008 4:22 PM >>>>>>>>> To: Apache AXIS C User List >>>>>>>>> Subject: RE: Core dumps in Axis libraries >>>>>>>>> >>>>>>>>> Hi Raghav, >>>>>>>>> >>>>>>>>> Can you please try this with --enable-guthilla=yes, and send the >>>>>>>>> dump.log, >>>>>>>>> and also the appropriate log_files generated by the >>>>>>>>> server/client. >>>>>>>>> >>>>>>>>> There also can be issues with your "condition variable" being >>>>>>>>> used >>>>>>>>> in >>>>>>>>> multiple threads. >>>>>>>>> >>>>>>>>> It also seems that if you pause between your requests you might >>>>>>>>> get >>>>>>>>> this >>>>>>>>> solved. According to what you say, when you increase the >>>>>>>>> log_level, >>>>>>>>> obviously you waste some time there which might be the solution >>>>>>>>> rather. >>>>>>>>> You can use the AXIS2_SLEEP() and AXIS2_USLEEP() inside Axis2/C >>>>>>>>> to >>>>>>>>> pause >>>>>>>>> between requests. You can get an idea on how to use these >>>>>>>>> functions >>>>>>>>> by >>>>>>>>> taking a look at the Axis2/C user_guide/client samples. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Senaka >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> As suggested, I tried using axis2c-1.3.0 both with and without >>>>>>>>>> --enable-guthilla=yes. But I still see different dumps >>>>>>>>>> consistently. >>>>>>>>>> Please find the attached file with back-traces. >>>>>>>>>> >>>>>>>>>> Are you using the same service client instance from multiple >>>>>>>>>> threads? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>>>> We have a axis thread pool with default number of threads and >>>>>>>>>>>>> a >>>>>>>>>>>>> set >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>> of our own threads(about 10, lets call them "database threads"). >>>>>>>>>> + On receiving a HTTP soap request, axis thread invokes the >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> appropriate >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> function that is bound to our logic. >>>>>>>>>> + Now, the axis thread requests a "database thread" to perform >>>>>>>>>> the >>>>>>>>>> appropriate query and waits on a condition variable. >>>>>>>>>> + our "db thread" performs the necessary query and signals on >>>>>>>>>> that >>>>>>>>>> condition variable so that axis thread shall take the control >>>>>>>>>> over. >>>>>>>>>> + axis thread will create the om output and dispatch as usual. >>>>>>>>>> >>>>>>>>>> My doubts: >>>>>>>>>> + does this design has any apparent drawback when there is a >>>>>>>>>> flurry >>>>>>>>>> of >>>>>>>>>> back-to-back soap requests? >>>>>>>>>> + Is there a need to increase the axis thread pool size? If yes, >>>>>>>>>> how >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> to >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> do it? >>>>>>>>>> + As told before, the dump is seen during a particular scenario >>>>>>>>>> only. >>>>>>>>>> i.e: while trying to add a list of already existing users to my >>>>>>>>>> database. For each user there is a SOAP request generated and >>>>>>>>>> the >>>>>>>>>> requests are sequential, i.e the second request is sent only >>>>>>>>>> after >>>>>>>>>> the >>>>>>>>>> response for previous request is received. >>>>>>>>>> + if I increase the log level of my own module that uses axis2c, >>>>>>>>>> there >>>>>>>>>> is no dump. On increasing the log level, my module writes quite >>>>>>>>>> a >>>>>>>>>> bit >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> of >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> information to my log file. >>>>>>>>>> >>>>>>>>>> Your help is appreciated. >>>>>>>>>> >>>>>>>>>> Thanks & Regards, >>>>>>>>>> ~raghav >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -----Original Message----- >>>>>>>>>> From: Samisa Abeysinghe [mailto:[EMAIL PROTECTED] >>>>>>>>>> Sent: Wednesday, March 05, 2008 7:25 PM >>>>>>>>>> To: Apache AXIS C User List >>>>>>>>>> Subject: Re: Core dumps in Axis libraries >>>>>>>>>> >>>>>>>>>> Raghavendra SM wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Thanks Manjula, >>>>>>>>>>> >>>>>>>>>>> But thats not a viable option for us. We would still want to >>>>>>>>>>> continue >>>>>>>>>>> using axis2C-1.0 for some more time. >>>>>>>>>>> >>>>>>>>>>> Are these dumps well known? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> No. >>>>>>>>>> >>>>>>>>>> Are you using the same service client instance from multiple >>>>>>>>>> threads? >>>>>>>>>> >>>>>>>>>> Samisa... >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>> To unsubscribe, e-mail: [EMAIL PROTECTED] >>>>>>>>>> For additional commands, e-mail: [EMAIL PROTECTED] >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>> To unsubscribe, e-mail: [EMAIL PROTECTED] >>>>>>>>>> For additional commands, e-mail: [EMAIL PROTECTED] >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> --------------------------------------------------------------------- >>>>>>>>> To unsubscribe, e-mail: [EMAIL PROTECTED] >>>>>>>>> For additional commands, e-mail: [EMAIL PROTECTED] >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> --------------------------------------------------------------------- >>>>>>>>> To unsubscribe, e-mail: [EMAIL PROTECTED] >>>>>>>>> For additional commands, e-mail: [EMAIL PROTECTED] >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> -- >>>>>>>> Samisa Abeysinghe >>>>>>>> Software Architect; WSO2 Inc. >>>>>>>> >>>>>>>> http://www.wso2.com/ - "Oxygenating the Web Service Platform." >>>>>>>> >>>>>>>> >>>>>>>> --------------------------------------------------------------------- >>>>>>>> To unsubscribe, e-mail: [EMAIL PROTECTED] >>>>>>>> For additional commands, e-mail: [EMAIL PROTECTED] >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> --------------------------------------------------------------------- >>>>>>> To unsubscribe, e-mail: [EMAIL PROTECTED] >>>>>>> For additional commands, e-mail: [EMAIL PROTECTED] >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> -- >>>>>> Samisa Abeysinghe >>>>>> Software Architect; WSO2 Inc. >>>>>> >>>>>> http://www.wso2.com/ - "Oxygenating the Web Service Platform." >>>>>> >>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe, e-mail: [EMAIL PROTECTED] >>>>>> For additional commands, e-mail: [EMAIL PROTECTED] >>>>>> >>>>>> >>>>>> >>>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: [EMAIL PROTECTED] >>>>> For additional commands, e-mail: [EMAIL PROTECTED] >>>>> >>>>> >>>>> >>>>> >>>>> >>>> -- >>>> Samisa Abeysinghe >>>> Software Architect; WSO2 Inc. >>>> >>>> http://www.wso2.com/ - "Oxygenating the Web Service Platform." >>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: [EMAIL PROTECTED] >>>> For additional commands, e-mail: [EMAIL PROTECTED] >>>> >>>> >>>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: [EMAIL PROTECTED] >>> For additional commands, e-mail: [EMAIL PROTECTED] >>> >>> >>> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [EMAIL PROTECTED] >> For additional commands, e-mail: [EMAIL PROTECTED] >> >> >> >> > > > -- > Samisa Abeysinghe > Software Architect; WSO2 Inc. > > http://www.wso2.com/ - "Oxygenating the Web Service Platform." > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
