Re: Very critical: Memory leak in freeradius-1.1.6
nikitha george wrote: On 5/23/07, nikitha george [EMAIL PROTECTED] wrote: Please find the valgrind output below. It shows so much memory is still reachable. That's because the server doesn't clean up memory on exit. Run it with the -m flag on the command line, and it will try to clean up memory on exit. I guess we are not cleaning up the all the expired cached session at regular interval. You are the only person running FreeRADIUS who is seeing this. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: Very critical: Memory leak in freeradius-1.1.6
On 5/23/07, nikitha george [EMAIL PROTECTED] wrote: Please find the valgrind output below. It shows so much memory is still reachable. I guess we are not cleaning up the all the expired cached session at regular interval. ==21844== 7,456 bytes in 29 blocks are still reachable in loss record 33 of 44 ==21844==at 0x48054FB: realloc (vg_replace_malloc.c:306) ==21844==by 0x351D54: (within /lib/libcrypto.so.0.9.8b) ==21844==by 0x352486: CRYPTO_realloc (in /lib/libcrypto.so.0.9.8b) ==21844==by 0x3A4776: lh_insert (in /lib/libcrypto.so.0.9.8b) ==21844==by 0x355527: OBJ_NAME_add (in /lib/libcrypto.so.0.9.8b) ==21844==by 0x3AC41C: EVP_add_digest (in /lib/libcrypto.so.0.9.8b) ==21844==by 0x486EF91: SSL_library_init (in /lib/libssl.so.0.9.8b) ==21844==by 0x4BAAE03: eaptls_attach (rlm_eap_tls.c:287) ==21844==by 0x4B95230: eaptype_load (eap.c:122) ==21844==by 0x4B93D1B: eap_instantiate (rlm_eap.c:145) ==21844==by 0xCCBE: find_module_instance (modules.c:358) ==21844==by 0xDCBD: do_compile_modsingle (modcall.c:1005) ==21844== ==21844== ==21844== 10,692 bytes in 33 blocks are still reachable in loss record 34 of 44 ==21844==at 0x4805400: malloc (vg_replace_malloc.c:149) ==21844==by 0x4830106: pairmake (valuepair.c:1049) ==21844==by 0x4830A58: pairread (valuepair.c:1244) ==21844==by 0x4830C15: userparse (valuepair.c:1296) ==21844==by 0x9BAB: pairlist_read (files.c:200) ==21844==by 0x4BBB5FF: preprocess_instantiate (rlm_preprocess.c:493) ==21844==by 0xCCBE: find_module_instance (modules.c:358) ==21844==by 0xDCBD: do_compile_modsingle (modcall.c:1005) ==21844==by 0xD34C: setup_modules (modules.c:580) ==21844==by 0x10A35: main (radiusd.c:965) ==21844== ==21844== ==21844== 13,325 bytes in 21 blocks are still reachable in loss record 35 of 44 ==21844==at 0x480473F: calloc (vg_replace_malloc.c:279) ==21844==by 0x4FE8F57A: _dl_new_object (in /lib/ld-2.5.so) ==21844==by 0x4FE8B0E0: _dl_map_object_from_fd (in /lib/ld-2.5.so) ==21844==by 0x4FE8D403: _dl_map_object (in /lib/ld-2.5.so) ==21844==by 0x4FE96668: dl_open_worker (in /lib/ld-2.5.so) ==21844==by 0x4FE92C05: _dl_catch_error (in /lib/ld-2.5.so) ==21844==by 0x4FE96191: _dl_open (in /lib/ld-2.5.so) ==21844==by 0x419BCD0C: dlopen_doit (in /lib/libdl-2.5.so) ==21844==by 0x4FE92C05: _dl_catch_error (in /lib/ld-2.5.so) ==21844==by 0x419BD38B: _dlerror_run (in /lib/libdl-2.5.so) ==21844==by 0x419BCC43: dlopen@@GLIBC_2.1 (in /lib/libdl-2.5.so) ==21844==by 0x48392A9: sys_dl_open (ltdl.c:958) ==21844== ==21844== ==21844== 15,808 bytes in 670 blocks are still reachable in loss record 36 of 44 ==21844==at 0x4805400: malloc (vg_replace_malloc.c:149) ==21844==by 0x418C001F: strdup (in /lib/libc-2.5.so) ==21844==by 0x79D7: cf_section_read (conffile.c:207) ==21844==by 0x8094: conf_read (conffile.c:917) ==21844==by 0xB55D: read_radius_conf_file (mainconfig.c:1264) ==21844==by 0xB6A5: read_mainconfig (mainconfig.c:1309) ==21844==by 0x109F2: main (radiusd.c:941) ==21844== ==21844== ==21844== 26,768 bytes in 336 blocks are still reachable in loss record 37 of 44 ==21844==at 0x4805400: malloc (vg_replace_malloc.c:149) ==21844==by 0x4B944E1: eap_compose (eap.c:395) ==21844==by 0x4B93AC8: eap_authenticate (rlm_eap.c:341) ==21844==by 0xE3C7: modcall (modcall.c:236) ==21844==by 0xEA6B: call_one (modcall.c:269) ==21844==by 0xE5B9: modcall (modcall.c:324) ==21844==by 0xC63D: indexed_modcall (modules.c:469) ==21844==by 0x5213: rad_check_password (auth.c:380) ==21844==by 0x579A: rad_authenticate (auth.c:675) ==21844==by 0xFC66: rad_respond (radiusd.c:1675) ==21844==by 0x116B1: main (radiusd.c:1440) ==21844== ==21844== ==21844== 49,152 bytes in 4 blocks are still reachable in loss record 38 of 44 ==21844==at 0x4805400: malloc (vg_replace_malloc.c:149) ==21844==by 0x4825B3F: lrad_hash_table_insert (hash.c:375) ==21844==by 0x4822AAF: dict_addattr (dict.c:478) ==21844==by 0x482316B: my_dict_init (dict.c:744) ==21844==by 0x4822F71: my_dict_init (dict.c:1050) ==21844==by 0x4822F71: my_dict_init (dict.c:1050) ==21844==by 0x4823DC5: dict_init (dict.c:1258) ==21844==by 0xB5AF: read_radius_conf_file (mainconfig.c:1276) ==21844==by 0xB6A5: read_mainconfig (mainconfig.c:1309) ==21844==by 0x109F2: main (radiusd.c:941) ==21844== ==21844== ==21844== 64,892 bytes in 1,704 blocks are still reachable in loss record 39 of 44 ==21844==at 0x4805400: malloc (vg_replace_malloc.c:149) ==21844==by 0x153DC: rad_malloc (util.c:308) ==21844==by 0x79AE: cf_section_read (conffile.c:203) ==21844==by 0x8094: conf_read (conffile.c:917) ==21844==by 0xB55D: read_radius_conf_file (mainconfig.c:1264) ==21844==by 0xB6A5: read_mainconfig (mainconfig.c:1309) ==21844==by 0x109F2: main (radiusd.c:941) ==21844== ==21844== ==21844== 136,877 bytes in 5,331 blocks are still
Re: Very critical: Memory leak in freeradius-1.1.6
nikitha george wrote: On 5/20/07, Alan DeKok [EMAIL PROTECTED] wrote: If valgrind doesn't say that the memory is lost, then the memory is very likely still being used. i.e. It's likely needed for something. something..? Did you read my earlier explanations? Why do radiusd need to hold the memory even after cleaning up the request? Did you read my earlier message explaining this? Even after stopping the request, the memory never gets freed. Why is this strange result of huge memory usage by radiusd? Did you read my earlier message explaining this? Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: Very critical: Memory leak in freeradius-1.1.6
Do you have any idea what kind of malloc/free implementation on our system will work out in this scenario? Thanks, Nikitha On 5/19/07, Alan DeKok [EMAIL PROTECTED] wrote: nikitha george wrote: So why is it happening in my case then? I can see all the requests gets cleaned up ( log message was put) but still so much memory is consumed by radiusd. When the server caches the requests, it uses memory to do that. When it frees the requests, the memory does *not* get returned to the system. This is usually due to the malloc/free implementation on your system, and cannot be controlled by the server. You want me post the huge log file..? I badly need this fix now.. Configuration wise i am using the default configuration except users and clients entry added newly. If your server really is that busy, then it needs that much memory to ensure it maintains quality service. If you make it use less memory, requests will be lost, and your network will suffer. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: Very critical: Memory leak in freeradius-1.1.6
Hi Martin, Thanks for your reply. The memory usage of radiusd in our system went from 6687KB to 160MB which is 50.1% of memory of our system and radiusd got re-started. I had run the Valgrind to see the resource leak. Definitely lost memory was 380 Bytes but still reachable are more. I think this is the one which is leaking memory. I dont have access to the log files today, i will post it by tomorrow for your reference. Thanks, Nikitha On 5/19/07, Martin Gadbois [EMAIL PROTECTED] wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 nikitha george wrote: So why is it happening in my case then? I can see all the requests gets cleaned up ( log message was put) but still so much memory is consumed by radiusd. Memory usage under Linux is a tricky thing. It depends if you're looking at RSS (RES in top), which is the actual RAM used by the process, or virtual memory (VIRT in top) which actually may map directly to disk and/or to RAM, and/or be shared with another process! A typical usage of a process the RES will rise, and then hit a plateau. Check the memory usage after a day, it should be the same the following week. If the RES raise linearly for days until the system trashes, you may have detected a memory leak. But to make sure a leak happens, usually tools like Valgrind helps better than looking at what the kernel reports, as there is a lot more hidden than what shows. - -- == +-+ Martin Gadbois | Please answer by yes or no.| Sr. SW Designer| Uncooperative user waste precious CPU time | Colubris Networks Inc. | -- The Andromeda Strain, M. Crichton, 1969 | -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.5 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGThWW9Y3/iTTCEDkRAnOMAJ47LwlA9o48V/49KnahEtaaNWf+4QCfXTL1 HAza0Po1crhG3tHzf6OQIcY= =q2Gp -END PGP SIGNATURE- - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: Very critical: Memory leak in freeradius-1.1.6
nikitha george wrote: Hi Martin, Thanks for your reply. The memory usage of radiusd in our system went from 6687KB to 160MB which is 50.1% of memory of our system and radiusd got re-started. I had run the Valgrind to see the resource leak. Definitely lost memory was 380 Bytes but still reachable are more. I think this is the one which is leaking memory. I dont have access to the log files today, i will post it by tomorrow for your reference. You seem pretty sure you've got a memory leak. As Alan has said, we're pretty sure there are no major leaks in the server core, so logically it's either your system libraries or an infrequently-used bit of the server. We need to know/see the following: * What OS are you on? Versions of the OS, kernel, major libraries (libc, threading) and a rough idea of how many clients and requests/sec you're getting * your radiusd.conf * a debug output of a *few* (not thousands!) of requests from your server - run radiusd -X A couple of wild guesses: Are you using PAM? Are you using rlm_perl or rlm_python? Are you using a database that's a bit too slow? I've you've actually run the valgrind --memcheck --leak-check=yes then let's take a look at that too. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: Very critical: Memory leak in freeradius-1.1.6
nikitha george wrote: The memory usage of radiusd in our system went from 6687KB to 160MB which is 50.1% of memory of our system and radiusd got re-started. 160Mb does sound like a lot. I had run the Valgrind to see the resource leak. Definitely lost memory was 380 Bytes but still reachable are more. I think this is the one which is leaking memory. I dont have access to the log files today, i will post it by tomorrow for your reference. The 380 bytes definitely lost is from libltdl. It contributed almost nothing to the 160M total memory used by the server. If valgrind doesn't say that the memory is lost, then the memory is very likely still being used. i.e. It's likely needed for something. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: Very critical: Memory leak in freeradius-1.1.6
On 5/20/07, Alan DeKok [EMAIL PROTECTED] wrote: If valgrind doesn't say that the memory is lost, then the memory is very likely still being used. i.e. It's likely needed for something. something..? Why do radiusd need to hold the memory even after cleaning up the request? Even after stopping the request, the memory never gets freed. Why is this strange result of huge memory usage by radiusd? On 5/20/07, Alan DeKok [EMAIL PROTECTED] wrote: nikitha george wrote: The memory usage of radiusd in our system went from 6687KB to 160MB which is 50.1% of memory of our system and radiusd got re-started. 160Mb does sound like a lot. I had run the Valgrind to see the resource leak. Definitely lost memory was 380 Bytes but still reachable are more. I think this is the one which is leaking memory. I dont have access to the log files today, i will post it by tomorrow for your reference. The 380 bytes definitely lost is from libltdl. It contributed almost nothing to the 160M total memory used by the server. If valgrind doesn't say that the memory is lost, then the memory is very likely still being used. i.e. It's likely needed for something. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: Very critical: Memory leak in freeradius-1.1.6
nikitha george wrote: I am seeing a very serious memory leak issue with freeradius-1.1.6. The memory usage of freeradius gone from 3386Byte to 64MB when i was trying to connect 16 clients with roaming interval of 1 second. More Access-Requests are coming and we keep saving those requests until cleanup_delay. Yes... that's how the server is supposed to work. After my initial investigation in the souce code, we keep cleaning the requests only if the select() fails, No. It cleans the requests after it's examined all of the sockets for data. which means no more request to handle. But in my case the clients are keep sending the request i think the cached request_list is not cleared properly or misses out some requests or something happens i guess. I may be wrong too, so please let me know what could be the root cause. No. People are running the server with sustain loads of 100's of packets per second. There is no memory leak like you say. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: Very critical: Memory leak in freeradius-1.1.6
So why is it happening in my case then? I can see all the requests gets cleaned up ( log message was put) but still so much memory is consumed by radiusd. You want me post the huge log file..? I badly need this fix now.. Configuration wise i am using the default configuration except users and clients entry added newly. Hoping to get some positive reply. Thanks, Nikitha On 5/18/07, Alan DeKok [EMAIL PROTECTED] wrote: nikitha george wrote: I am seeing a very serious memory leak issue with freeradius-1.1.6. The memory usage of freeradius gone from 3386Byte to 64MB when i was trying to connect 16 clients with roaming interval of 1 second. More Access-Requests are coming and we keep saving those requests until cleanup_delay. Yes... that's how the server is supposed to work. After my initial investigation in the souce code, we keep cleaning the requests only if the select() fails, No. It cleans the requests after it's examined all of the sockets for data. which means no more request to handle. But in my case the clients are keep sending the request i think the cached request_list is not cleared properly or misses out some requests or something happens i guess. I may be wrong too, so please let me know what could be the root cause. No. People are running the server with sustain loads of 100's of packets per second. There is no memory leak like you say. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: Very critical: Memory leak in freeradius-1.1.6
nikitha george wrote: So why is it happening in my case then? I can see all the requests gets cleaned up ( log message was put) but still so much memory is consumed by radiusd. When the server caches the requests, it uses memory to do that. When it frees the requests, the memory does *not* get returned to the system. This is usually due to the malloc/free implementation on your system, and cannot be controlled by the server. You want me post the huge log file..? I badly need this fix now.. Configuration wise i am using the default configuration except users and clients entry added newly. If your server really is that busy, then it needs that much memory to ensure it maintains quality service. If you make it use less memory, requests will be lost, and your network will suffer. Alan DeKok. -- http://deployingradius.com - The web site of the book http://deployingradius.com/blog/ - The blog - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: Very critical: Memory leak in freeradius-1.1.6
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 nikitha george wrote: So why is it happening in my case then? I can see all the requests gets cleaned up ( log message was put) but still so much memory is consumed by radiusd. Memory usage under Linux is a tricky thing. It depends if you're looking at RSS (RES in top), which is the actual RAM used by the process, or virtual memory (VIRT in top) which actually may map directly to disk and/or to RAM, and/or be shared with another process! A typical usage of a process the RES will rise, and then hit a plateau. Check the memory usage after a day, it should be the same the following week. If the RES raise linearly for days until the system trashes, you may have detected a memory leak. But to make sure a leak happens, usually tools like Valgrind helps better than looking at what the kernel reports, as there is a lot more hidden than what shows. - -- == +-+ Martin Gadbois | Please answer by yes or no.| Sr. SW Designer| Uncooperative user waste precious CPU time | Colubris Networks Inc. | -- The Andromeda Strain, M. Crichton, 1969 | -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.5 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGThWW9Y3/iTTCEDkRAnOMAJ47LwlA9o48V/49KnahEtaaNWf+4QCfXTL1 HAza0Po1crhG3tHzf6OQIcY= =q2Gp -END PGP SIGNATURE- - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Very critical: Memory leak in freeradius-1.1.6
Hi, I am seeing a very serious memory leak issue with freeradius-1.1.6. The memory usage of freeradius gone from 3386Byte to 64MB when i was trying to connect 16 clients with roaming interval of 1 second. More Access-Requests are coming and we keep saving those requests until cleanup_delay. After my initial investigation in the souce code, we keep cleaning the requests only if the select() fails, which means no more request to handle. But in my case the clients are keep sending the request i think the cached request_list is not cleared properly or misses out some requests or something happens i guess. I may be wrong too, so please let me know what could be the root cause. Awaiting for any earliest reply. Thanks, Nikitha - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
