Yes, I do agree regarding the threads spawned. The kernel assigns about 2MB -stack space- to every thread upon it's creation. Further, because (as I took from the link below) every thread can see the stack of any neighbour, this necessarily means that the total of all stacks must be present at any time. The identical VSZ for the threads is an indication here.
Btw : my Oracle shows such effects. It seems to pull gigantic amounts for each thread. But this is a misinterpretation. --------- However and regrettably, for my newpki_server the case is different. There are no <n> LinuxThreads. The newpki_server spawns exactly 1 thread. This one pulls 700MB and more. If there were more threads, I would see that in ps -lax. According to your demo program and the docu below generally under Linux any thread is treated as a heavy weight procces and gets a processNumber assigned. Because its not, I conclude there must be something other under the hood. I find this discussion on "bug or feature" quite relevant. At least for myself it will guide my decision to set newpki_server productive or not. Wolf useful link : http://pauillac.inria.fr/~xleroy/linuxthreads/faq.html -----Urspr�ngliche Nachricht----- Von: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Auftrag von Fr�d�ric Giudicelli Gesendet: Freitag, 30. Januar 2004 14:45 An: [EMAIL PROTECTED] Betreff: Re: Many instances of newpki-server running Once again this is not a problem, it's pthread implementation. As for apache, they didn't use threads only for security reasons; apache spawns processes, so that if one them segfaults, the master process can re-spawn a new one, but it ends up being exactly the same thing regarding memory usage: on my system I have 12 children processes, each of them taking 20884kB, and the master process takes 15500kb, for a totall of 260mB of virtual memory. This virtual memory is not used, so it's not a problem. Let's not spend days arguing about it since it's only a pthread matter, NewPKI doesn't use much memoty, as I said previously around 10mB. [EMAIL PROTECTED] wrote: > Yes, I could verify this. Threads are spawned with VSZ=185816 kB. The swap > is not touched. The question now is, if that is a neglectable flaw or > something that becomes nasty some day on other aspects. Anyway, it still > deserves more investigation because VSZ problems are no common problems of > standard software under Linux. For example, the apache gurus (apache 1.3.27 > source) refrained from pthreads and made use of pthread_mutex only. This is > stuff for more investigation. > > -----Urspr�ngliche Nachricht----- > Von: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] Auftrag von Fr�d�ric Giudicelli > Gesendet: Freitag, 30. Januar 2004 09:04 > An: [EMAIL PROTECTED] > Betreff: Re: Many instances of newpki-server running > > > Well, after browsing google for a while, it seems like this is normal, > as apparently pthreads uses a lot of virtual memory but never commits it. > Here is why, each thread is created with a stack of 8388608B, so 90 * > 8388608B = 720mB. > > There, you have your explaintation :) > > Fr�d�ric Giudicelli wrote: > > >>Oops, never mind, the code I sent is wrong, it should be: >>for(i=0; i<90; i++) >>instead of >>for(i=0; i<50; i++) >> >>With the 90 threads, which is pretty much the number I have with my 8 >>entities, I get a VSZ of 739060kB, so mysql is not involved in this >>mess, it's all pthread's fault! :) >> >> >>Fr�d�ric Giudicelli wrote: >> >> >>>Ok, so after a lot of investigastions, I came to this conclusion: >>>either pthread sucks or VSZ is no relevent.... >>> >>>Here is a little testing program (test.cpp): >>> >>>#include <unistd.h> >>>#include <pthread.h> >>>#include <stdio.h> >>> >>>void * tmp_ThreadProc(void * Param) >>>{ >>> for(;;) >>> { >>> usleep(1000000); >>> printf("%d\n", pthread_self()); >>> } >>>} >>> >>>int main(int argc, char ** argv) >>>{ >>> pthread_attr_t attr; >>> pthread_t m_thread_id[90]; >>> int i; >>> >>> pthread_attr_init(&attr); >>> pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_JOINABLE); >>> >>> for(i=0; i<50; i++) >>> { >>> if(pthread_create(&m_thread_id[i], &attr, &tmp_ThreadProc, NULL)) >>> { >>> return 1; >>> } >>> } >>> pthread_attr_destroy(&attr); >>> >>> pthread_join(m_thread_id[0], NULL); >>> >>> return 0; >>>} >>> >>> >>>To compile: >>>gcc -o test -lpthread test.cpp >>> >>>After running it, on my system, it showed a VSZ of over 400mB.... >>>There is nothing I can do if pthread has a weird implementation... >>> >>>The extra mB from the usual 700, are comming from libmysql, it seems >>>to be using pthread as well, I'm testing it right now to see if there >>>are any memory leaks comming from the way I use mysql. >>> >>>[EMAIL PROTECTED] wrote: >>> >>> >>>>Hi Frederic, >>>> >>>>VSZ versus /proc/meminfo : >>>> >>>>/proc/meminf does not show the virt mem consumption because the mem was >>>>malloced - but not used. The kernel is intelligent enough to just make a >>>>reservation without assigning it when the malloced mem is not >>>>referenced. >>>> >>>>So newpki_server seems to malloc or sbrk without using the mem. >>>> >>>>For example, the prog below mallocs VSZ=200MB of virtual mem. Without >>>>the >>>>memset below you cannot see effect in /proc/meminfo when the prog is >>>>running. >>>> >>>> >>>>>From my standpoint newpki_server misbehaves in that regard. >>>> >>>> >>>> >>>>Wolf >>>>---------------- >>>> >>>>#include <unistd.h> >>>>#include <stdio.h> >>>> >>>>main () { >>>> >>>> char c; >>>> void *startof; >>>> >>>> startof = sbrk (1024*1024*200); >>>> memset (startof,'a',1024*1024*200); /* just this bangs into the swap */ >>>> read (0,&c,1); >>>>} >>>> >>>> >>>>-----Urspr�ngliche Nachricht----- >>>>Von: [EMAIL PROTECTED] >>>>[mailto:[EMAIL PROTECTED] Auftrag von Fr�d�ric Giudicelli >>>>Gesendet: Donnerstag, 29. Januar 2004 01:51 >>>>An: [EMAIL PROTECTED] >>>>Betreff: Re: Many instances of newpki-server running >>>> >>>> >>>>Ok, after a long investigation, here are the conclusions: there is no >>>>memory leak and VSZ really is virtual :). >>>> >>>>Let me explain, for every thread, pthread "adds" 32mB to the vm. >>>>I did this test, I reduced to 1 the number of connection threads >>>>(instead of the 50 orignal ones) and bingo the VSZ reduced to a value >>>>around 100mB, instead of the initial 700mB. >>>>I then re-enabled the 50 connection threads, recompiled newpki_server, >>>>and ran 'cat /proc/meminfo' before running newpki_server, here is the >>>>output: >>>> total: used: free: >>>>Mem: 517894144 482705408 35188736 >>>>Swap: 534601728 26198016 508403712 >>>> >>>>I then ran newpki_server (with 8 entities in it), waited a bit and >>>>re-ran 'cat /proc/meminfo', here is the output >>>> total: used: free: >>>>Mem: 517894144 486785024 31109120 >>>>Swap: 534601728 26198016 508403712 >>>> >>>>As you can see the swap didn't move, however 'cat /proc/`pidof >>>>newpki_server`/status' tells me: >>>>VmSize: 766168 kB >>>> >>>>As you can see, the info of VSZ is not to be taken for granted :) >>>> >>>>If you really want to do the test, install and run newpki_server on a >>>>Win2000 server, configure it to have as many entities as in you linux >>>>box, once the server is up and running run the "task manager", and look >>>>at the memory usage of newpki_server.exe (in Windows the memory usage is >>>>for the ram and the swap, there is no "bullshit" about its meaning), you >>>>will see, it should be around 10mB. >>>> >>>>Regards, >>>>-- >>>>Fr�d�ric Giudicelli >>>>http://www.newpki.org >>>> >>>>_____________________________________________________________________ >>>>NewPKI http://www.newpki.org >>>>User Support Mailing List [EMAIL PROTECTED] >>>>Automated List Manager [EMAIL PROTECTED] >>>> >>>>_____________________________________________________________________ >>>>NewPKI http://www.newpki.org >>>>User Support Mailing List [EMAIL PROTECTED] >>>>Automated List Manager [EMAIL PROTECTED] >>> >>> >>> > > -- > Fr�d�ric Giudicelli > http://www.newpki.org > _____________________________________________________________________ > NewPKI http://www.newpki.org > User Support Mailing List [EMAIL PROTECTED] > Automated List Manager [EMAIL PROTECTED] > > _____________________________________________________________________ > NewPKI http://www.newpki.org > User Support Mailing List [EMAIL PROTECTED] > Automated List Manager [EMAIL PROTECTED] -- Fr�d�ric Giudicelli http://www.newpki.org _____________________________________________________________________ NewPKI http://www.newpki.org User Support Mailing List [EMAIL PROTECTED] Automated List Manager [EMAIL PROTECTED] _____________________________________________________________________ NewPKI http://www.newpki.org User Support Mailing List [EMAIL PROTECTED] Automated List Manager [EMAIL PROTECTED]
