RE: Why does Apache do this braindamaged dlclose/dlopen stuff?
So in the longer term, is there a reason the parent has to contain the interpreter at all? Can't it just do a system call when it needs one? It seems a bit excessive to put aside a couple of megabytes of system memory just to run startup.pl. Well, remember that the interpreter itself will remain shared throughout, so there's no real disadvantage in having in the parent. The main reason to run startup.pl in the parent is to overcome as much of Perl's startup time as possible. Compiling the code domainates the startup time, so the thing to do is to pull in your modules in startup.pl . That way, it's only done once, and the results are shared between all children. I think the thing to do here is fix the memory leaks 8-) Stephen.
RE: Why does Apache do this braindamaged dlclose/dlopen stuff?
Hi there, On Fri, 21 Jan 2000, Stephen Anderson wrote: So in the longer term, is there a reason the parent has to contain the interpreter at all? Can't it just do a system call when it needs one? Well, remember that the interpreter itself will remain shared throughout, so there's no real disadvantage in having in the parent. I thought that if the parent was light it could replace its proxy, which would save a lot of messing about. I think the thing to do here is fix the memory leaks 8-) Can't argue with that. 73, Ged.
RE: Why does Apache do this braindamaged dlclose/dlopen stuff?
Hi all, On Wed, 19 Jan 2000, Gerald Richter wrote: in the long term, the solution that you have prefered in previous mail, not to unload modperl at all, maybe the better one As I understand it with Apache/mod_perl: 1. The parent (contains the Perl interpreter) fires up, initialises things and launches some children. Any memory it leaks stays leaked until restart. That could be weeks away. Apart from making copies of it, most of the time it doesn't do much with the interpreter. More waste. 2. The children occasionally get the coup de grace, so we recover any memory they leaked. They do lots with the interpreter. 3. When the parent fork()s a new child it can fork some leaked memory too, which gradually will become unshared, so the longer this goes on the closer we get to bringing the whole system to its knees. So in the longer term, is there a reason the parent has to contain the interpreter at all? Can't it just do a system call when it needs one? It seems a bit excessive to put aside a couple of megabytes of system memory just to run startup.pl. If one could get around any process communication difficulties, the children could be just the same as they are now, but exec()ed instead of fork()ed by a (smaller) child process which has never leaked any memory. The exec() latency isn't an issue because of the way that Apache preforks a pool of processes and the overhead will be minimal if the children live long enough. Please tell me if I have got this all around my neck. 73, Ged.
RE: Why does Apache do this braindamaged dlclose/dlopen stuff?
-Original Message- From: Gerald Richter [mailto:[EMAIL PROTECTED]] Sent: 19 January 2000 04:36 To: Alan Burlison; [EMAIL PROTECTED] Subject: RE: Why does Apache do this braindamaged dlclose/dlopen stuff? So I would agree to your last sentences that Dynloader is responsible for unloading, because that's the only module, which knows what it had loaded. Agreed. It's a relatively small change to DynaLoader, with great benefits for embedded Perl. Also I am not so sure if unloading all the libraries can be really successfully done, because the Perl XS libraries don't assume that they will ever unloaded (because they are actualy only unloaded when the program exits). This may be the reason for memory leaks Daniel metioned earlier, because the XS libraries don't have a chance to free there resources (or not aware of that they actually should do so). Yes and no. If XS libraries are written with OO-style wrappers (which, IMHO, they always should be), then surely you can catch the unloading in a DESTROY sub and use that to do the deallocation of resources? Perl can only manage Perl resources, and extension resources should be the responsibility of the the programmer. Stephen.
Re: Why does Apache do this braindamaged dlclose/dlopen stuff?
On Tue, Jan 18, 2000 at 08:03:42PM +, Alan Burlison wrote: I think the correct fix is for the Apache core to avoid dlclosing anything it has dlopened in the first place. If new modules have been added to the config files, they should be dlopened, but any old ones should *not* be dlclosed, EVEN IF THEY ARE NO LONGER IN THE CONFIG FILE!!! I believe they did that on purpose, akin to the line let's see if we survive a complete restart as via apachectl restart. -- Jens-Uwe Mager HELIOS Software GmbH Steinriede 3 30827 Garbsen Germany Phone: +49 5131 709320 FAX:+49 5131 709325 Internet: [EMAIL PROTECTED]
Re: Why does Apache do this braindamaged dlclose/dlopen stuff?
On Tue, Jan 18, 2000 at 12:59:13PM +, Alan Burlison wrote: Can someone please explain why Apache does all the dlclosing and dlopening of shared files on startup and a restart? I can think of no reson why this would ever be necessary - why on earth is it done? Alan Burlison Probably the biggest reason for dlopen/dlclose on a restart is that the list of modules in the config file can change on a restart. The reason for the reload on startup has something to do with parsing the config file in the parent and child; it was never adequately explained to me. The trick would be getting it not to do this without busting up the module API, which I can actually think of a few ways to do, and in a way that the Apache Group didn't rigorously object to :) Dan /\ /\ | Daniel Jacobowitz|__|SCS Class of 2002 | | Debian GNU/Linux Developer__Carnegie Mellon University | | [EMAIL PROTECTED] | | [EMAIL PROTECTED] | \/ \/
RE: Why does Apache do this braindamaged dlclose/dlopen stuff?
Can someone please explain why Apache does all the dlclosing and dlopening of shared files on startup and a restart? I can think of no reson why this would ever be necessary - why on earth is it done? I don't know, but I know for sure that causes a lot of trouble with mod_perl and Perl modules which uses XS code... Gerald - Gerald Richterecos electronic communication services gmbh Internetconnect * Webserver/-design/-datenbanken * Consulting Post: Tulpenstrasse 5 D-55276 Dienheim b. Mainz E-Mail: [EMAIL PROTECTED] Voice:+49 6133 925151 WWW:http://www.ecos.de Fax: +49 6133 925152 -
RE: Why does Apache do this braindamaged dlclose/dlopen stuff?
Probably the biggest reason for dlopen/dlclose on a restart is that the list of modules in the config file can change on a restart. The reason for the reload on startup has something to do with parsing the config file in the parent and child; it was never adequately explained to me. The trick would be getting it not to do this without busting up the module API, which I can actually think of a few ways to do, and in a way that the Apache Group didn't rigorously object to :) Do you know, would it be possible to catch the dlclose inside mod_perl and unload (dlclose) all external libaries that Perl has loaded so far? The problem is, that these libraries will persist between such an dlclose/dlopen and afterward don't get proberly initialised, so some of the code really points to data from the old Perl interpreter, which causes strange behaviour. Gerald - Gerald Richterecos electronic communication services gmbh Internetconnect * Webserver/-design/-datenbanken * Consulting Post: Tulpenstrasse 5 D-55276 Dienheim b. Mainz E-Mail: [EMAIL PROTECTED] Voice:+49 6133 925151 WWW:http://www.ecos.de Fax: +49 6133 925152 -
Re: Why does Apache do this braindamaged dlclose/dlopen stuff?
On Tue, Jan 18, 2000 at 02:40:59PM +0100, Gerald Richter wrote: Probably the biggest reason for dlopen/dlclose on a restart is that the list of modules in the config file can change on a restart. The reason for the reload on startup has something to do with parsing the config file in the parent and child; it was never adequately explained to me. The trick would be getting it not to do this without busting up the module API, which I can actually think of a few ways to do, and in a way that the Apache Group didn't rigorously object to :) Do you know, would it be possible to catch the dlclose inside mod_perl and unload (dlclose) all external libaries that Perl has loaded so far? The problem is, that these libraries will persist between such an dlclose/dlopen and afterward don't get proberly initialised, so some of the code really points to data from the old Perl interpreter, which causes strange behaviour. That is what my patch did. And that was the explanation I posted of the problem last week when we were debugging it. Dan /\ /\ | Daniel Jacobowitz|__|SCS Class of 2002 | | Debian GNU/Linux Developer__Carnegie Mellon University | | [EMAIL PROTECTED] | | [EMAIL PROTECTED] | \/ \/
RE: Why does Apache do this braindamaged dlclose/dlopen stuff?
That is what my patch did. And that was the explanation I posted of the problem last week when we were debugging it. Sorry, I missed that thread. I have posted this problem more then once here, because it have beaten me and other often when using Embperl. The problem there is often more hidden, because it doesn't SIGSEGV, it still works, but some functionality (where Perl variables are tied to C variables) doesn't work, so it's often not easy to detect. Unfortunably I never had the time to track this down enought to create a real usefull patch (just a workaround, (PERL_STARTUP_DONE_CHECK), which will cause the XS libraries only loaded after the second load of libperl.so; this works for the startup, but not after a restart). Also I didn't tried it yet, your patch makes much sense too me. I will try it out as soon as I get a litle free time. The next step is to port it NT, because there isn't a dlopen/dlclose (of course there is one, but with a different name) and on NT Apache only work with dynamic libraries. If it works on Unix and NT it's should go straight to the CVS and a mod_perl 1.22 ! Thanks for solving that issue! Gerald - Gerald Richterecos electronic communication services gmbh Internetconnect * Webserver/-design/-datenbanken * Consulting Post: Tulpenstrasse 5 D-55276 Dienheim b. Mainz E-Mail: [EMAIL PROTECTED] Voice:+49 6133 925151 WWW:http://www.ecos.de Fax: +49 6133 925152 -
Re: Why does Apache do this braindamaged dlclose/dlopen stuff?
On Tue, Jan 18, 2000 at 08:26:17PM +0100, Gerald Richter wrote: Also I didn't tried it yet, your patch makes much sense too me. I will try it out as soon as I get a litle free time. The next step is to port it NT, because there isn't a dlopen/dlclose (of course there is one, but with a different name) and on NT Apache only work with dynamic libraries. If it works on Unix and NT it's should go straight to the CVS and a mod_perl 1.22 ! Now if only we could get the memory leaks tracked down... ltrace logs of apache are simply too huge to deal with, but I'll try anyway to straighten them out and figure what is not getting freed. Dan /\ /\ | Daniel Jacobowitz|__|SCS Class of 2002 | | Debian GNU/Linux Developer__Carnegie Mellon University | | [EMAIL PROTECTED] | | [EMAIL PROTECTED] | \/ \/
Re: Why does Apache do this braindamaged dlclose/dlopen stuff?
Vivek Khera wrote: AB To summarise: Apache dlclose's the mod_perl.so, which then results in AB the perl libperl.so being unloaded as well (there's a linker dependency Excellent summary... thanks! AB from mod_perl - perl libperl.so). Unfortunately the perl XS modules AB loaded in during startup via dlopen are *not* unloaded, nor do they AB succeed in locking the perl libperl.so into memory (you could construe AB this as a linker bug). Then Apache reloads the mod_perl libperl.so, I think the linker is in error here for not adding the dependency on the library and that is what should be fixed... Don't worry, I already have a bug report open and someone from the linker group is having a look. I havn't been able to replicate the exact same problem in a minimal set of C files, but I'm working on it. I have a linker debug trace that shows the problem, though (on Solaris, use "LD_DEBUG=files,detail httpd -X" and look for the addresses that the perl libperl.so is mapped in at). However, other folks have reported the exact same problem on other OSs, eg Linux BSD, so I think that in the short term we need to be realistic and find a way of not tickling what seems to be a common linker bug. Alan Burlison
Re: Why does Apache do this braindamaged dlclose/dlopen stuff?
On Tue, Jan 18, 2000 at 08:03:42PM +, Alan Burlison wrote: The current fix is to forcibly unload the perl XS modules during the unload. However, on reflection I'm not at all sure this is the correct thing to do. Although you can unload the .so component of a perl module, you can't unload the .pm component, so just removing the .so part as in the current workaround is suspect at least. Remember - mod_perl is being unloaded. Perl going away. At this point perl_destruct/perl_free have already been called, and thus the .pm components are effectively unloaded. I think the correct fix is for the Apache core to avoid dlclosing anything it has dlopened in the first place. If new modules have been added to the config files, they should be dlopened, but any old ones should *not* be dlclosed, EVEN IF THEY ARE NO LONGER IN THE CONFIG FILE!!! I firmly believe this needs fixing in the Apache core, not by hacking around it in mod_perl. Alan Burlison Dan /\ /\ | Daniel Jacobowitz|__|SCS Class of 2002 | | Debian GNU/Linux Developer__Carnegie Mellon University | | [EMAIL PROTECTED] | | [EMAIL PROTECTED] | \/ \/
RE: Why does Apache do this braindamaged dlclose/dlopen stuff?
To summarise:... Thanks for the summary, but I already know this problem for a long time and I am very happy that somebody has taken the time track this down and provide a solution :-) However, other folks have reported the exact same problem on other OSs, eg Linux BSD, so I think that in the short term we need to be realistic and find a way of not tickling what seems to be a common linker bug. That's the same on NT. It seems to occur on all OSs, so it won't help anything to make the linker responsible, there are to much linkers... and I am not sure if the linker can know under all circumstances which libraries to unload. I don't think we will convice the Apache people in the short term, that they shouldn't unload the libraries (also we can discuss it with them). The only practical solution I see, is the one to unload the libraries, as the patch already does. The thing left to do is to port it to other OS's (like NT) which does not have a function named dlclose (we need simply use another name). This solution will work and if the Apache people one time decide to not unload the modules, it won't hurt. Gerald P.S. Does you get any feedback from your post to p5p about the unload function in Dynaloader?
Re: Why does Apache do this braindamaged dlclose/dlopen stuff?
Daniel Jacobowitz wrote: On Tue, Jan 18, 2000 at 08:03:42PM +, Alan Burlison wrote: The current fix is to forcibly unload the perl XS modules during the unload. However, on reflection I'm not at all sure this is the correct thing to do. Although you can unload the .so component of a perl module, you can't unload the .pm component, so just removing the .so part as in the current workaround is suspect at least. Remember - mod_perl is being unloaded. Perl going away. At this point perl_destruct/perl_free have already been called, and thus the .pm components are effectively unloaded. Ah yes, so they are. I still think dlclosing for no good reason sucks though. Alan Burlison
Re: Why does Apache do this braindamaged dlclose/dlopen stuff?
Gerald Richter wrote: That's the same on NT. It seems to occur on all OSs, so it won't help anything to make the linker responsible, there are to much linkers... and I am not sure if the linker can know under all circumstances which libraries to unload. Yes it can. Its main job is to keep track and control the dependencies between libraries. It's just that sometimes thy don't do a particularly good job of it ;-) I don't think we will convice the Apache people in the short term, that they shouldn't unload the libraries (also we can discuss it with them). The only practical solution I see, is the one to unload the libraries, as the patch already does. The thing left to do is to port it to other OS's (like NT) which does not have a function named dlclose (we need simply use another name). This solution will work and if the Apache people one time decide to not unload the modules, it won't hurt. I think they should be persuaded - this is a very insiduous bug and extremely hard to find. P.S. Does you get any feedback from your post to p5p about the unload function in Dynaloader? No. Nothing meaningful. Alan Burlison
RE: Why does Apache do this braindamaged dlclose/dlopen stuff?
Yes it can. No it can't :-) Its main job is to keep track and control the dependencies between libraries. It's just that sometimes thy don't do a particularly good job of it ;-) This works only if this dependencies are know at link time, but look at the source of Dynloader. You can retrieve address of any (public)symbol inside a library dynamicly at runtime. Now you have the entry address and can pass it around. No linker will ever have a chance to know what you do in your programm. As soon as you use such things (and Dynloader uses them), the linker doesn't have chance! Gerald
Re: Why does Apache do this braindamaged dlclose/dlopen stuff?
On Tue, Jan 18, 2000 at 10:19:04PM +0100, Gerald Richter wrote: Yes it can. No it can't :-) Its main job is to keep track and control the dependencies between libraries. It's just that sometimes thy don't do a particularly good job of it ;-) This works only if this dependencies are know at link time, but look at the source of Dynloader. You can retrieve address of any (public)symbol inside a library dynamicly at runtime. Now you have the entry address and can pass it around. No linker will ever have a chance to know what you do in your programm. As soon as you use such things (and Dynloader uses them), the linker doesn't have chance! You're confusing the dynamic and static linkers. The dynamic linker is what he was referring to; it knows what libraries it resolves symbols to. Dan /\ /\ | Daniel Jacobowitz|__|SCS Class of 2002 | | Debian GNU/Linux Developer__Carnegie Mellon University | | [EMAIL PROTECTED] | | [EMAIL PROTECTED] | \/ \/
RE: Why does Apache do this braindamaged dlclose/dlopen stuff?
You're confusing the dynamic and static linkers. The dynamic linker is what he was referring to; it knows what libraries it resolves symbols to. Yes, I know this difference and you will be right in most cases, but the address that is returned, could be passed around to other libraries and the linker can't know this. (the dynloader.so can retrieve an adresse of embperl.so and pass it's address to modperl.so, which then pass the address to whatever. How should the dynamic linker know, that whatever is calling the symbol in embperl.so (and stores the address) Gerald
Re: Why does Apache do this braindamaged dlclose/dlopen stuff?
On Tue, Jan 18, 2000 at 10:43:09PM +0100, Gerald Richter wrote: You're confusing the dynamic and static linkers. The dynamic linker is what he was referring to; it knows what libraries it resolves symbols to. Yes, I know this difference and you will be right in most cases, but the address that is returned, could be passed around to other libraries and the linker can't know this. (the dynloader.so can retrieve an adresse of embperl.so and pass it's address to modperl.so, which then pass the address to whatever. How should the dynamic linker know, that whatever is calling the symbol in embperl.so (and stores the address) Ah, I see what you mean. That's boundedly undefined - the linker certainly shouldn't be trying to catch that sort of case. But it should, IMO, catch bindings that it makes itself. Dan /\ /\ | Daniel Jacobowitz|__|SCS Class of 2002 | | Debian GNU/Linux Developer__Carnegie Mellon University | | [EMAIL PROTECTED] | | [EMAIL PROTECTED] | \/ \/
Re: Why does Apache do this braindamaged dlclose/dlopen stuff?
Gerald Richter wrote: This works only if this dependencies are know at link time, but look at the source of Dynloader. You can retrieve address of any (public)symbol inside a library dynamicly at runtime. Now you have the entry address and can pass it around. No linker will ever have a chance to know what you do in your programm. As soon as you use such things (and Dynloader uses them), the linker doesn't have chance! Nope, that's not how it works. Take a look at http://docs.sun.com:80/ab2/coll.45.10/LLM/@Ab2PageView/5121 *All* symbols in a shared library are known by ld.so Alan Burlison
Re: Why does Apache do this braindamaged dlclose/dlopen stuff?
Alan Burlison wrote: AB from mod_perl - perl libperl.so). Unfortunately the perl XS modules AB loaded in during startup via dlopen are *not* unloaded, nor do they AB succeed in locking the perl libperl.so into memory (you could construe AB this as a linker bug). Then Apache reloads the mod_perl libperl.so, Actually, I retract that statement. It is *not* a linker bug. By explicitly adding a dependency between the XS .so modules and the perl libperl.so, the problem can be made to dissapear, as ld.so then knows that there is a dependency between the XS module and the perl libperl.so I think the linker is in error here for not adding the dependency on the library and that is what should be fixed... Nope, the *programmer* (or in this case, MakeMaker) is in error for not specifying the dependency when the XS .so module was built. Here's what one of our linker developers has to say about the subject: [You may be assuming that] ii) the action of dlopen()'ing an object (say B.so) from an object (say A.so) creats a binding between the two objects that insure the latter cannot be deleted before the formed is deleted. This isn't the case. ld.so.1 maintains bindings/refence counts etc. between the objects it has control over - ie. a relocation in one object that references another. The dlopen() family are thought of as user code, and we do not try to interprete how objects are bound that use these facilities. In fact we can get in all sorts of issues if we do - a handle, just like a function pointer, can be handed to other objects, thus who knows what object *expects* another object to be in existance. i.e. if you decide to use dlopen/dlclose and you screw it up, then it is *your* fault, not the linkers. The fact that so many different systems show the same behaviour suggests that this is a common linker design decision. I actually think that the real fault is in DynaLoader. If you assume that the exiting of the perl interpreter is the same thing as the exit of the program, then it makes sense not to dlclose anything, as it will all be reclaimed on exit anyway. However, if you embed perl inside an application, the exit of the perl interpreter is certainly *not* the same thing as the program exiting, and DynaLoader should explicily dlclose all the files it has dlopened. I'm going to submit this as a bug to p5p. Alan Burlison
RE: Why does Apache do this braindamaged dlclose/dlopen stuff?
Actually, I retract that statement. It is *not* a linker bug. By explicitly adding a dependency between the XS .so modules and the perl libperl.so, the problem can be made to dissapear, as ld.so then knows that there is a dependency between the XS module and the perl libperl.so I think the linker is in error here for not adding the dependency on the library and that is what should be fixed... Nope, the *programmer* (or in this case, MakeMaker) is in error for not specifying the dependency when the XS .so module was built. Here's what one of our linker developers has to say about the subject: [You may be assuming that] ii) the action of dlopen()'ing an object (say B.so) from an object (say A.so) creats a binding between the two objects that insure the latter cannot be deleted before the formed is deleted. This isn't the case. ld.so.1 maintains bindings/refence counts etc. between the objects it has control over - ie. a relocation in one object that references another. The dlopen() family are thought of as user code, and we do not try to interprete how objects are bound that use these facilities. In fact we can get in all sorts of issues if we do - a handle, just like a function pointer, can be handed to other objects, thus who knows what object *expects* another object to be in existance. i.e. if you decide to use dlopen/dlclose and you screw it up, then it is *your* fault, not the linkers. The fact that so many different systems show the same behaviour suggests that this is a common linker design decision. That's exactly what I tried to explain in my last few mails, also I am not a linker guru :-) I actually think that the real fault is in DynaLoader. If you assume that the exiting of the perl interpreter is the same thing as the exit of the program, then it makes sense not to dlclose anything, as it will all be reclaimed on exit anyway. However, if you embed perl inside an application, the exit of the perl interpreter is certainly *not* the same thing as the program exiting, and DynaLoader should explicily dlclose all the files it has dlopened. I'm going to submit this as a bug to p5p. You are talking about two things, the first are the dependencies MakeMaker could add at link time. This will make dlopen/dlclose know what to unload, but, as the rest of Perl, Dynaloader is very "dynamicly" and let's you specify additional dependencies during loadtime (e.g. via the *.bs files). This can't be catched by MakeMaker. So I would agree to your last sentences that Dynloader is responsible for unloading, because that's the only module, which knows what it had loaded. Also I am not so sure if unloading all the libraries can be really successfully done, because the Perl XS libraries don't assume that they will ever unloaded (because they are actualy only unloaded when the program exits). This may be the reason for memory leaks Daniel metioned earlier, because the XS libraries don't have a chance to free there resources (or not aware of that they actually should do so). So in the long term, the solution that you have prefered in previous mail, not to unload modperl at all, maybe the better one, to stay compatible with the existing software. Gerald - Gerald Richterecos electronic communication services gmbh Internetconnect * Webserver/-design/-datenbanken * Consulting Post: Tulpenstrasse 5 D-55276 Dienheim b. Mainz E-Mail: [EMAIL PROTECTED] Voice:+49 6133 925151 WWW:http://www.ecos.de Fax: +49 6133 925152 -
