As for apache, they didn't use threads only for security reasons; apache spawns processes, so that if one them segfaults, the master process can re-spawn a new one, but it ends up being exactly the same thing regarding memory usage: on my system I have 12 children processes, each of them taking 20884kB, and the master process takes 15500kb, for a totall of 260mB of virtual memory.
This virtual memory is not used, so it's not a problem. Let's not spend days arguing about it since it's only a pthread matter, NewPKI doesn't use much memoty, as I said previously around 10mB.
[EMAIL PROTECTED] wrote:
Yes, I could verify this. Threads are spawned with VSZ=185816 kB. The swap is not touched. The question now is, if that is a neglectable flaw or something that becomes nasty some day on other aspects. Anyway, it still deserves more investigation because VSZ problems are no common problems of standard software under Linux. For example, the apache gurus (apache 1.3.27 source) refrained from pthreads and made use of pthread_mutex only. This is stuff for more investigation.
-----Urspr�ngliche Nachricht----- Von: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Auftrag von Fr�d�ric Giudicelli Gesendet: Freitag, 30. Januar 2004 09:04 An: [EMAIL PROTECTED] Betreff: Re: Many instances of newpki-server running
Well, after browsing google for a while, it seems like this is normal, as apparently pthreads uses a lot of virtual memory but never commits it. Here is why, each thread is created with a stack of 8388608B, so 90 * 8388608B = 720mB.
There, you have your explaintation :)
Fr�d�ric Giudicelli wrote:
Oops, never mind, the code I sent is wrong, it should be: for(i=0; i<90; i++) instead of for(i=0; i<50; i++)
With the 90 threads, which is pretty much the number I have with my 8 entities, I get a VSZ of 739060kB, so mysql is not involved in this mess, it's all pthread's fault! :)
Fr�d�ric Giudicelli wrote:
Ok, so after a lot of investigastions, I came to this conclusion: either pthread sucks or VSZ is no relevent....
Here is a little testing program (test.cpp):
#include <unistd.h> #include <pthread.h> #include <stdio.h>
void * tmp_ThreadProc(void * Param) { for(;;) { usleep(1000000); printf("%d\n", pthread_self()); } }
int main(int argc, char ** argv) { pthread_attr_t attr; pthread_t m_thread_id[90]; int i;
pthread_attr_init(&attr); pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_JOINABLE);
for(i=0; i<50; i++) { if(pthread_create(&m_thread_id[i], &attr, &tmp_ThreadProc, NULL)) { return 1; } } pthread_attr_destroy(&attr);
pthread_join(m_thread_id[0], NULL);
return 0; }
To compile: gcc -o test -lpthread test.cpp
After running it, on my system, it showed a VSZ of over 400mB.... There is nothing I can do if pthread has a weird implementation...
The extra mB from the usual 700, are comming from libmysql, it seems to be using pthread as well, I'm testing it right now to see if there are any memory leaks comming from the way I use mysql.
[EMAIL PROTECTED] wrote:
Hi Frederic,
VSZ versus /proc/meminfo :
/proc/meminf does not show the virt mem consumption because the mem was malloced - but not used. The kernel is intelligent enough to just make a reservation without assigning it when the malloced mem is not referenced.
So newpki_server seems to malloc or sbrk without using the mem.
For example, the prog below mallocs VSZ=200MB of virtual mem. Without the memset below you cannot see effect in /proc/meminfo when the prog is running.
From my standpoint newpki_server misbehaves in that regard.
Wolf ----------------
#include <unistd.h> #include <stdio.h>
main () {
char c; void *startof;
startof = sbrk (1024*1024*200); memset (startof,'a',1024*1024*200); /* just this bangs into the swap */ read (0,&c,1); }
-----Urspr�ngliche Nachricht----- Von: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Auftrag von Fr�d�ric Giudicelli Gesendet: Donnerstag, 29. Januar 2004 01:51 An: [EMAIL PROTECTED] Betreff: Re: Many instances of newpki-server running
Ok, after a long investigation, here are the conclusions: there is no memory leak and VSZ really is virtual :).
Let me explain, for every thread, pthread "adds" 32mB to the vm. I did this test, I reduced to 1 the number of connection threads (instead of the 50 orignal ones) and bingo the VSZ reduced to a value around 100mB, instead of the initial 700mB. I then re-enabled the 50 connection threads, recompiled newpki_server, and ran 'cat /proc/meminfo' before running newpki_server, here is the output: total: used: free: Mem: 517894144 482705408 35188736 Swap: 534601728 26198016 508403712
I then ran newpki_server (with 8 entities in it), waited a bit and re-ran 'cat /proc/meminfo', here is the output total: used: free: Mem: 517894144 486785024 31109120 Swap: 534601728 26198016 508403712
As you can see the swap didn't move, however 'cat /proc/`pidof newpki_server`/status' tells me: VmSize: 766168 kB
As you can see, the info of VSZ is not to be taken for granted :)
If you really want to do the test, install and run newpki_server on a Win2000 server, configure it to have as many entities as in you linux box, once the server is up and running run the "task manager", and look at the memory usage of newpki_server.exe (in Windows the memory usage is for the ram and the swap, there is no "bullshit" about its meaning), you will see, it should be around 10mB.
Regards, -- Fr�d�ric Giudicelli http://www.newpki.org
_____________________________________________________________________ NewPKI http://www.newpki.org User Support Mailing List [EMAIL PROTECTED] Automated List Manager [EMAIL PROTECTED]
_____________________________________________________________________ NewPKI http://www.newpki.org User Support Mailing List [EMAIL PROTECTED] Automated List Manager [EMAIL PROTECTED]
-- Fr�d�ric Giudicelli http://www.newpki.org _____________________________________________________________________ NewPKI http://www.newpki.org User Support Mailing List [EMAIL PROTECTED] Automated List Manager [EMAIL PROTECTED]
_____________________________________________________________________ NewPKI http://www.newpki.org User Support Mailing List [EMAIL PROTECTED] Automated List Manager [EMAIL PROTECTED]
-- Fr�d�ric Giudicelli http://www.newpki.org _____________________________________________________________________ NewPKI http://www.newpki.org User Support Mailing List [EMAIL PROTECTED] Automated List Manager [EMAIL PROTECTED]
