Hi Rene, Have you had any further success based on the email advise from Andreas Fink below and the Kannel configurations and valgrind results from Hanh, on tracking down further memory leaks in smppbox?
We all know finding memory leaks is not easy. Links for Valgrind report and Kannel Configuration from Hanh: 1. https://www.dropbox.com/s/wbnkewitosiwv5i/opensmppbox_memleakcheck_20140502.txt 2. Here is my kannel configuration: https://www.dropbox.com/s/u90vs8srbubbmhn/kannel_configuration_file.txt Thanks From: Andreas Fink [mailto:af...@list.fink.org] Sent: Sunday, May 4, 2014 1:47 PM To: hbil...@ecommunicate.biz Subject: Re: Does opensmppbox and smsbox have serious memory issues? well valgrind gives you the hints you need to find them as it tells you where the remaining stuff is allocated (and usually somewhere its forgotten to be released). I'm not actively using smppbox as I have my own front end so I was just spotting it by looking at your logs. So lets take a look what valgrind tells us: We look for leaks which happen for every sms. ==23650== Memcheck, a memory error detector ==23650== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al. ==23650== Using Valgrind-3.9.0 and LibVEX; rerun with -h for copyright info ==23650== Command: /usr/local/sbin/opensmppbox -v -d -- /etc/kannel/opensmppbox.conf ==23650== Parent PID: 23648 ==23650== ==23650== ==23650== HEAP SUMMARY: ==23650== in use at exit: 2,213,708 bytes in 25,777 blocks ==23650== total heap usage: 688,951 allocs, 663,174 frees, 227,552,203 bytes allocated ==23650== ==23650== 48 (32 direct, 16 indirect) bytes in 2 blocks are definitely lost in loss record 457 of 714 ==23650== at 0x4027434: malloc (vg_replace_malloc.c:291) ==23650== by 0x80978C3: gw_native_malloc (gwmem-native.c:87) ==23650== by 0x80A3FB1: octstr_create_from_data_real (octstr.c:263) ==23650== by 0x80A4591: octstr_duplicate_real (octstr.c:377) ==23650== by 0x8056DC6: smpp_to_bearerbox (opensmppbox.c:1569) <-- this is duplicate an octstr which never gets released. only twice so not critical ==23650== by 0x8098BBE: new_thread (gwthread-pthread.c:385) ==23650== by 0x46F9C38: start_thread (pthread_create.c:304) ==23650== by 0x482F78D: clone (clone.S:130) ==23650== ==23650== 49 (32 direct, 17 indirect) bytes in 2 blocks are definitely lost in loss record 458 of 714 ==23650== at 0x4027434: malloc (vg_replace_malloc.c:291) ==23650== by 0x80978C3: gw_native_malloc (gwmem-native.c:87) ==23650== by 0x80A3FB1: octstr_create_from_data_real (octstr.c:263) ==23650== by 0x80A4591: octstr_duplicate_real (octstr.c:377) ==23650== by 0x808B846: cfg_get_real (cfg.c:670) <-- this is probably a one time leak of something read from the config file ==23650== by 0x8052839: main (opensmppbox.c:2292) ==23650== ==23650== 144 bytes in 1 blocks are possibly lost in loss record 519 of 714 ==23650== at 0x402574D: calloc (vg_replace_malloc.c:618) ==23650== by 0x40111FB: _dl_allocate_tls (dl-tls.c:300) ==23650== by 0x46FA5A0: pthread_create@@GLIBC_2.1 (allocatestack.c:580) ==23650== by 0x4412F2A: my_thread_global_init (in /usr/lib/i386-linux-gnu/libmysqlclient.so.18.0.0) ==23650== by 0x44112F7: my_init (in /usr/lib/i386-linux-gnu/libmysqlclient.so.18.0.0) <-- this would be something mysql client library allocates. I presume some one time leak ==23650== by 0x43EC93A: mysql_server_init (in /usr/lib/i386-linux-gnu/libmysqlclient.so.18.0.0) ==23650== by 0x43EE078: mysql_init (in /usr/lib/i386-linux-gnu/libmysqlclient.so.18.0.0) ==23650== by 0x8095752: mysql_open_conn (dbpool_mysql.c:84) ==23650== by 0x8095AD5: dbpool_increase (dbpool.c:200) ==23650== by 0x8095C06: dbpool_create (dbpool.c:166) ==23650== by 0x805B3D3: dlr_init_mysql (dlr_mysql.c:452) ==23650== by 0x8058BA3: dlr_init (dlr.c:257) ==23650== ==23650== 164 (80 direct, 84 indirect) bytes in 1 blocks are definitely lost in loss record 531 of 714 ==23650== at 0x4027434: malloc (vg_replace_malloc.c:291) ==23650== by 0x80978C3: gw_native_malloc (gwmem-native.c:87) ==23650== by 0x80A15FF: gwlist_create_real (list.c:131) ==23650== by 0x80600DC: sms_split (sms.c:339) <--- some long sms getting split leaving one block behind? ==23650== by 0x805520E: run_smppbox (opensmppbox.c:1008) ==23650== by 0x8098BBE: new_thread (gwthread-pthread.c:385) ==23650== by 0x46F9C38: start_thread (pthread_create.c:304) ==23650== by 0x482F78D: clone (clone.S:130) ==23650== ==23650== 450 (288 direct, 162 indirect) bytes in 18 blocks are definitely lost in loss record 589 of 714 ==23650== at 0x4027434: malloc (vg_replace_malloc.c:291) ==23650== by 0x80978C3: gw_native_malloc (gwmem-native.c:87) ==23650== by 0x80A3FB1: octstr_create_from_data_real (octstr.c:263) ==23650== by 0x80A4591: octstr_duplicate_real (octstr.c:377) ==23650== by 0x80547DD: run_smppbox (opensmppbox.c:2106) <--- this is duplicating a string. ==23650== by 0x8098BBE: new_thread (gwthread-pthread.c:385) ==23650== by 0x46F9C38: start_thread (pthread_create.c:304) ==23650== by 0x482F78D: clone (clone.S:130) ==23650== ==23650== 1,460 (1,184 direct, 276 indirect) bytes in 74 blocks are definitely lost in loss record 654 of 714 ==23650== at 0x4027434: malloc (vg_replace_malloc.c:291) ==23650== by 0x80978C3: gw_native_malloc (gwmem-native.c:87) ==23650== by 0x80A3FB1: octstr_create_from_data_real (octstr.c:263) ==23650== by 0x80A4591: octstr_duplicate_real (octstr.c:377) ==23650== by 0x805DB4D: msg_duplicate (msg-decl.h:80) ==23650== by 0x80565AD: catenate_msg (opensmppbox.c:525) <-- this is duplicating a message. probably to reconcatenate. This has the potentail to eat lots of memory. ==23650== by 0x80568FB: check_multipart (opensmppbox.c:1481) ==23650== by 0x8057B70: smpp_to_bearerbox (opensmppbox.c:1639) ==23650== by 0x8098BBE: new_thread (gwthread-pthread.c:385) ==23650== by 0x46F9C38: start_thread (pthread_create.c:304) ==23650== by 0x482F78D: clone (clone.S:130) ==23650== ==23650== 1,555 (1,200 direct, 355 indirect) bytes in 75 blocks are definitely lost in loss record 656 of 714 ==23650== at 0x4027434: malloc (vg_replace_malloc.c:291) ==23650== by 0x80978C3: gw_native_malloc (gwmem-native.c:87) ==23650== by 0x80A3FB1: octstr_create_from_data_real (octstr.c:263) ==23650== by 0x80A4591: octstr_duplicate_real (octstr.c:377) ==23650== by 0x805D97F: msg_duplicate (msg-decl.h:80) ==23650== by 0x80565AD: catenate_msg (opensmppbox.c:525) <-- same ==23650== by 0x80568FB: check_multipart (opensmppbox.c:1481) ==23650== by 0x8057B70: smpp_to_bearerbox (opensmppbox.c:1639) ==23650== by 0x8098BBE: new_thread (gwthread-pthread.c:385) ==23650== by 0x46F9C38: start_thread (pthread_create.c:304) ==23650== by 0x482F78D: clone (clone.S:130) ==23650== ==23650== 2,160 bytes in 15 blocks are possibly lost in loss record 663 of 714 ==23650== at 0x402574D: calloc (vg_replace_malloc.c:618) ==23650== by 0x40111FB: _dl_allocate_tls (dl-tls.c:300) ==23650== by 0x46FA5A0: pthread_create@@GLIBC_2.1 (allocatestack.c:580) ==23650== by 0x809875F: gwthread_create_real (gwthread-pthread.c:475) ==23650== by 0x8053067: main (opensmppbox.c:2157) ==23650== ==23650== 2,304 bytes in 16 blocks are possibly lost in loss record 666 of 714 ==23650== at 0x402574D: calloc (vg_replace_malloc.c:618) ==23650== by 0x40111FB: _dl_allocate_tls (dl-tls.c:300) ==23650== by 0x46FA5A0: pthread_create@@GLIBC_2.1 (allocatestack.c:580) ==23650== by 0x809875F: gwthread_create_real (gwthread-pthread.c:475) ==23650== by 0x8054843: run_smppbox (opensmppbox.c:2125) ==23650== by 0x8098BBE: new_thread (gwthread-pthread.c:385) ==23650== by 0x46F9C38: start_thread (pthread_create.c:304) ==23650== by 0x482F78D: clone (clone.S:130) ==23650== ==23650== LEAK SUMMARY: ==23650== definitely lost: 2,816 bytes in 172 blocks ==23650== indirectly lost: 910 bytes in 165 blocks ==23650== possibly lost: 4,608 bytes in 32 blocks ==23650== still reachable: 2,205,374 bytes in 25,408 blocks ==23650== suppressed: 0 bytes in 0 blocks ==23650== Reachable blocks (those to which a pointer was found) are not shown. ==23650== To see them, rerun with: --leak-check=full --show-leak-kinds=all ==23650== ==23650== For counts of detected and suppressed errors, rerun with: -v ==23650== ERROR SUMMARY: 9 errors from 9 contexts (suppressed: 45 from 10) so so far not much leaks. I would suggest some issues with splitting /concatenating messages could be the issue. How is your setup configured in regards to concatenated sms? Kind Regards Hillel Bilman Manager eCommunicate mailto: hbil...@ecommunicate.co.za Cell: 083-2300002 Landline: 011-443-6164 Fax: 088-011-443-6164 Social Media marketing campaigns (Facebook, Google+, Twitter, LinkedIn etc) – Interactive Websites - .mobi Sites – Mobile Apps(Android, iPhone, Blackberry, Nokia) - Premium Rated SMSs and short codes - SMS competitions and campaigns – Lead Generation - opt-in subscription Billing – MMS campaigns - USSD campaigns - WAP - Outlook SMS – Bulk SMS and Bulk Email – Email 2 SMS 2 Email - Developer Kit for Mobile Services integration - Voice Over IP services From: Hanh Le Bich [mailto:hanhmi...@gmail.com] Sent: 20 May 2014 04:54 AM To: hbil...@ecommunicate.biz Cc: devel@kannel.org; Rene Kluwen Subject: Re: Does opensmppbox and smsbox have serious memory issues? Hello again, Sorry for late reply, i was stuck on field. 1. I upload again the previous Valgrind report in below link also in attachment as usual: https://www.dropbox.com/s/wbnkewitosiwv5i/opensmppbox_memleakcheck_20140502.txt 2. Here is my kannel configuration: https://www.dropbox.com/s/u90vs8srbubbmhn/kannel_configuration_file.txt **********************************kannel.conf**************************** group = core admin-port = 13000 smsbox-port = 13001 admin-password = ****** status-password = ****** log-file = "/var/log/kannel/bearerbox.log" access-log = "/var/log/kannel/bearerbox-access.log" log-level = 1 ssl-server-cert-file = "/etc/kannel/cert.pem" ssl-server-key-file = "/etc/kannel/key.pem" dlr-storage = mysql admin-allow-ip = "*.*.*.*" smsbox-max-pending = 100 sms-incoming-queue-limit = -1 store-type = file store-location = "/var/log/kannel/kannel.store" store-dump-freq = 10 include = "/etc/kannel/smsc.conf" include = "/etc/kannel/service_route.conf" group = smsbox smsbox-id = Redundancy bearerbox-host = localhost sendsms-port = 13013 sendsms-chars = "0123456789 +-" log-file = "/var/log/kannel/smsbox.log" log-level = 2 #For DLR group = mysql-connection id = bb_dlr_id host = localhost username = root password = ***** database = kannel max-connections = 10 group = dlr-db id = bb_dlr_id table = kannel_bb_dlr field-smsc = smsc field-timestamp = ts field-destination = destination field-source = source field-service = service field-url = url field-mask = mask field-status = status field-boxc-id = boxc **********************************sqlbox.conf**************************** group = sqlbox id = sqlbox-idd smsbox-id = sqlbox bearerbox-host = localhost bearerbox-port = 13001 smsbox-port = 13005 smsbox-port-ssl = false limit-per-cycle = 50 save-dlr = 0 sql-log-table = sent_sms sql-insert-table = send_sms log-file = "/var/log/kannel/sqlbox.log" log-level = 1 group = mysql-connection id = sqlbox-idd host = localhost username = root password = ****** database = kannel max-connections = 10 **********************************opensmppbox.conf**************************** group = core dlr-storage = mysql group = opensmppbox opensmppbox-id = smppbox-idd opensmppbox-port = 2775 bearerbox-host = 127.0.0.1 bearerbox-port = 13005 our-system-id = thefirstsmppbox smpp-logins = /etc/kannel/smpplogins.txt use-systemid-as-smsboxid = 1 log-file = /var/log/kannel/smppbox.log log-level = 1 group = smsc-route smsc-id = main_smsc smsbox-id = payment shortcode = "155" group = smsc-route smsc-id = main_smsc smsbox-id = lumata shortcode = "146" group = mysql-connection id = smppboxdlr-idd host = localhost username = root password = ****** database = kannel max-connections = 10 group = dlr-db id = smppboxdlr-idd table = kannel_smppbox_dlr field-smsc = smsc field-timestamp = ts field-destination = destination field-source = source field-service = service field-url = url field-mask = mask field-status = status field-boxc-id = boxc **********************************service_route.conf**************************** group = smsbox-route smsbox-id = payment shortcode = "155" group = smsbox-route smsbox-id = lumata shortcode = "146" **********************************smsc.conf**************************** group = smsc smsc = smpp smsc-id = main_smsc allowed-smsc-id = main_smsc host = "x.x.x.x" port = 45289 log-file="/var/log/kannel/smsc.log" log-level = 1 transceiver-mode = true smsc-username = "SMSCGW" smsc-password = ****** system-type = "VMA" reconnect-delay = 10 enquire-link-interval = 300 throughput = 150 max-pending-submits = 200 max-sms-octets = 140 source-addr-ton = 0 source-addr-npi = 1 alt-charset = "ASCII" alt-addr-charset = "GSM" ---------------------------------------------------------- Regards, Hanh On Sun, May 18, 2014 at 2:19 PM, <hbil...@ecommunicate.biz> wrote: Hi Hahn, 1) Please will you resend your results for the latest version of opensmppbox from svn with valgrind --leak-check=full --track-origins=yes ./opensmppbox? The previous dropbox version of your results is no longer available at https://www.dropbox.com/s/wbnkewitosiwv5i/opensmppbox_memleakcheck_20140502 2) Please will you also send this email list your Kannel configuration. Hopefully Alex will solve the memory leak for smsbox and Rene will solve the memory leak for smppbox. thanks From: Andreas Fink [mailto:af...@list.fink.org] Sent: Sunday, May 4, 2014 1:47 PM To: hbil...@ecommunicate.biz Subject: Re: Does opensmppbox and smsbox have serious memory issues? well valgrind gives you the hints you need to find them as it tells you where the remaining stuff is allocated (and usually somewhere its forgotten to be released). I'm not actively using smppbox as I have my own front end so I was just spotting it by looking at your logs. So lets take a look what valgrind tells us: We look for leaks which happen for every sms. ==23650== Memcheck, a memory error detector ==23650== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al. ==23650== Using Valgrind-3.9.0 and LibVEX; rerun with -h for copyright info ==23650== Command: /usr/local/sbin/opensmppbox -v -d -- /etc/kannel/opensmppbox.conf ==23650== Parent PID: 23648 ==23650== ==23650== ==23650== HEAP SUMMARY: ==23650== in use at exit: 2,213,708 bytes in 25,777 blocks ==23650== total heap usage: 688,951 allocs, 663,174 frees, 227,552,203 bytes allocated ==23650== ==23650== 48 (32 direct, 16 indirect) bytes in 2 blocks are definitely lost in loss record 457 of 714 ==23650== at 0x4027434: malloc (vg_replace_malloc.c:291) ==23650== by 0x80978C3: gw_native_malloc (gwmem-native.c:87) ==23650== by 0x80A3FB1: octstr_create_from_data_real (octstr.c:263) ==23650== by 0x80A4591: octstr_duplicate_real (octstr.c:377) ==23650== by 0x8056DC6: smpp_to_bearerbox (opensmppbox.c:1569) <-- this is duplicate an octstr which never gets released. only twice so not critical ==23650== by 0x8098BBE: new_thread (gwthread-pthread.c:385) ==23650== by 0x46F9C38: start_thread (pthread_create.c:304) ==23650== by 0x482F78D: clone (clone.S:130) ==23650== ==23650== 49 (32 direct, 17 indirect) bytes in 2 blocks are definitely lost in loss record 458 of 714 ==23650== at 0x4027434: malloc (vg_replace_malloc.c:291) ==23650== by 0x80978C3: gw_native_malloc (gwmem-native.c:87) ==23650== by 0x80A3FB1: octstr_create_from_data_real (octstr.c:263) ==23650== by 0x80A4591: octstr_duplicate_real (octstr.c:377) ==23650== by 0x808B846: cfg_get_real (cfg.c:670) <-- this is probably a one time leak of something read from the config file ==23650== by 0x8052839: main (opensmppbox.c:2292) ==23650== ==23650== 144 bytes in 1 blocks are possibly lost in loss record 519 of 714 ==23650== at 0x402574D: calloc (vg_replace_malloc.c:618) ==23650== by 0x40111FB: _dl_allocate_tls (dl-tls.c:300) ==23650== by 0x46FA5A0: pthread_create@@GLIBC_2.1 (allocatestack.c:580) ==23650== by 0x4412F2A: my_thread_global_init (in /usr/lib/i386-linux-gnu/libmysqlclient.so.18.0.0) ==23650== by 0x44112F7: my_init (in /usr/lib/i386-linux-gnu/libmysqlclient.so.18.0.0) <-- this would be something mysql client library allocates. I presume some one time leak ==23650== by 0x43EC93A: mysql_server_init (in /usr/lib/i386-linux-gnu/libmysqlclient.so.18.0.0) ==23650== by 0x43EE078: mysql_init (in /usr/lib/i386-linux-gnu/libmysqlclient.so.18.0.0) ==23650== by 0x8095752: mysql_open_conn (dbpool_mysql.c:84) ==23650== by 0x8095AD5: dbpool_increase (dbpool.c:200) ==23650== by 0x8095C06: dbpool_create (dbpool.c:166) ==23650== by 0x805B3D3: dlr_init_mysql (dlr_mysql.c:452) ==23650== by 0x8058BA3: dlr_init (dlr.c:257) ==23650== ==23650== 164 (80 direct, 84 indirect) bytes in 1 blocks are definitely lost in loss record 531 of 714 ==23650== at 0x4027434: malloc (vg_replace_malloc.c:291) ==23650== by 0x80978C3: gw_native_malloc (gwmem-native.c:87) ==23650== by 0x80A15FF: gwlist_create_real (list.c:131) ==23650== by 0x80600DC: sms_split (sms.c:339) <--- some long sms getting split leaving one block behind? ==23650== by 0x805520E: run_smppbox (opensmppbox.c:1008) ==23650== by 0x8098BBE: new_thread (gwthread-pthread.c:385) ==23650== by 0x46F9C38: start_thread (pthread_create.c:304) ==23650== by 0x482F78D: clone (clone.S:130) ==23650== ==23650== 450 (288 direct, 162 indirect) bytes in 18 blocks are definitely lost in loss record 589 of 714 ==23650== at 0x4027434: malloc (vg_replace_malloc.c:291) ==23650== by 0x80978C3: gw_native_malloc (gwmem-native.c:87) ==23650== by 0x80A3FB1: octstr_create_from_data_real (octstr.c:263) ==23650== by 0x80A4591: octstr_duplicate_real (octstr.c:377) ==23650== by 0x80547DD: run_smppbox (opensmppbox.c:2106) <--- this is duplicating a string. ==23650== by 0x8098BBE: new_thread (gwthread-pthread.c:385) ==23650== by 0x46F9C38: start_thread (pthread_create.c:304) ==23650== by 0x482F78D: clone (clone.S:130) ==23650== ==23650== 1,460 (1,184 direct, 276 indirect) bytes in 74 blocks are definitely lost in loss record 654 of 714 ==23650== at 0x4027434: malloc (vg_replace_malloc.c:291) ==23650== by 0x80978C3: gw_native_malloc (gwmem-native.c:87) ==23650== by 0x80A3FB1: octstr_create_from_data_real (octstr.c:263) ==23650== by 0x80A4591: octstr_duplicate_real (octstr.c:377) ==23650== by 0x805DB4D: msg_duplicate (msg-decl.h:80) ==23650== by 0x80565AD: catenate_msg (opensmppbox.c:525) <-- this is duplicating a message. probably to reconcatenate. This has the potentail to eat lots of memory. ==23650== by 0x80568FB: check_multipart (opensmppbox.c:1481) ==23650== by 0x8057B70: smpp_to_bearerbox (opensmppbox.c:1639) ==23650== by 0x8098BBE: new_thread (gwthread-pthread.c:385) ==23650== by 0x46F9C38: start_thread (pthread_create.c:304) ==23650== by 0x482F78D: clone (clone.S:130) ==23650== ==23650== 1,555 (1,200 direct, 355 indirect) bytes in 75 blocks are definitely lost in loss record 656 of 714 ==23650== at 0x4027434: malloc (vg_replace_malloc.c:291) ==23650== by 0x80978C3: gw_native_malloc (gwmem-native.c:87) ==23650== by 0x80A3FB1: octstr_create_from_data_real (octstr.c:263) ==23650== by 0x80A4591: octstr_duplicate_real (octstr.c:377) ==23650== by 0x805D97F: msg_duplicate (msg-decl.h:80) ==23650== by 0x80565AD: catenate_msg (opensmppbox.c:525) <-- same ==23650== by 0x80568FB: check_multipart (opensmppbox.c:1481) ==23650== by 0x8057B70: smpp_to_bearerbox (opensmppbox.c:1639) ==23650== by 0x8098BBE: new_thread (gwthread-pthread.c:385) ==23650== by 0x46F9C38: start_thread (pthread_create.c:304) ==23650== by 0x482F78D: clone (clone.S:130) ==23650== ==23650== 2,160 bytes in 15 blocks are possibly lost in loss record 663 of 714 ==23650== at 0x402574D: calloc (vg_replace_malloc.c:618) ==23650== by 0x40111FB: _dl_allocate_tls (dl-tls.c:300) ==23650== by 0x46FA5A0: pthread_create@@GLIBC_2.1 (allocatestack.c:580) ==23650== by 0x809875F: gwthread_create_real (gwthread-pthread.c:475) ==23650== by 0x8053067: main (opensmppbox.c:2157) ==23650== ==23650== 2,304 bytes in 16 blocks are possibly lost in loss record 666 of 714 ==23650== at 0x402574D: calloc (vg_replace_malloc.c:618) ==23650== by 0x40111FB: _dl_allocate_tls (dl-tls.c:300) ==23650== by 0x46FA5A0: pthread_create@@GLIBC_2.1 (allocatestack.c:580) ==23650== by 0x809875F: gwthread_create_real (gwthread-pthread.c:475) ==23650== by 0x8054843: run_smppbox (opensmppbox.c:2125) ==23650== by 0x8098BBE: new_thread (gwthread-pthread.c:385) ==23650== by 0x46F9C38: start_thread (pthread_create.c:304) ==23650== by 0x482F78D: clone (clone.S:130) ==23650== ==23650== LEAK SUMMARY: ==23650== definitely lost: 2,816 bytes in 172 blocks ==23650== indirectly lost: 910 bytes in 165 blocks ==23650== possibly lost: 4,608 bytes in 32 blocks ==23650== still reachable: 2,205,374 bytes in 25,408 blocks ==23650== suppressed: 0 bytes in 0 blocks ==23650== Reachable blocks (those to which a pointer was found) are not shown. ==23650== To see them, rerun with: --leak-check=full --show-leak-kinds=all ==23650== ==23650== For counts of detected and suppressed errors, rerun with: -v ==23650== ERROR SUMMARY: 9 errors from 9 contexts (suppressed: 45 from 10) so so far not much leaks. I would suggest some issues with splitting /concatenating messages could be the issue. How is your setup configured in regards to concatenated sms? On 04 May 2014, at 13:02, hbil...@ecommunicate.biz wrote: Hi Andreas, Any ideas on where the remaining memory leaks can be found for opensmmppbox? Your last post on this issue fixed a major memory issue. Well done. rgds ---------------------------------------------------------------------- Date: Fri, 2 May 2014 22:47:29 +0700 From: Hanh Le Bich <hanhmi...@gmail.com> To: hbil...@ecommunicate.biz Cc: devel@kannel.org Subject: Re: Does opensmppbox and smsbox have serious memory issues? Message-ID: <ca+p4dgwahg-5zxtvunfao7aminqee3qkp7+furbpm7m2rm_...@mail.gmail.com> Content-Type: text/plain; charset="utf-8" Hello, I send the valgrind report following Mr. Rene advice: valgrind --leak-check=full --track-origins=yes ./opensmppbox I('ll send *smsbox* report by tomorrow) As my observation, the memory leak of opensmmppbox is significant decreased, even i haven't seen RAM consumption change for entire day, this is great. Refer to the report, still leaking, isn't it? ==23650== LEAK SUMMARY: ==23650== definitely lost: 2,816 bytes in 172 blocks ==23650== indirectly lost: 910 bytes in 165 blocks ==23650== possibly lost: 4,608 bytes in 32 blocks ==23650== still reachable: 2,205,374 bytes in 25,408 blocks ==23650== suppressed: 0 bytes in 0 blocks ==23650== Reachable blocks (those to which a pointer was found) are not shown. ==23650== To see them, rerun with: --leak-check=full --show-leak-kinds=all ==23650== ==23650== For counts of detected and suppressed errors, rerun with: -v ==23650== ERROR SUMMARY: 9 errors from 9 contexts (suppressed: 45 from 10) Also you can download the report at this link: https://www.dropbox.com/s/wbnkewitosiwv5i/opensmppbox_memleakcheck_20140502. txt Regards, Hanh. On Fri, May 2, 2014 at 9:33 PM, <hbil...@ecommunicate.biz> wrote: Hi Hand, Did you see the latest change by Rene committed to SVN ( http://www.kannel.org/pipermail/devel/2014-April/005612.html) Does it fix the memory issue you reported? rgds ---------------------------------------------------------------------- ---------------------------------------------------------------------- ---- Kind Regards Hillel Bilman Manager eCommunicate mailto: hbil...@ecommunicate.co.za Cell: 083-2300002 Landline: 011-443-6164 Fax: 088-011-443-6164 Social Media marketing campaigns (Facebook, Google+, Twitter, LinkedIn etc) ? Interactive Websites - .mobi Sites ? Mobile Apps(Android, iPhone, Blackberry, Nokia) - Premium Rated SMSs and short codes - SMS competitions and campaigns ? Lead Generation - opt-in subscription Billing ? MMS campaigns - USSD campaigns - WAP - Outlook SMS ? Bulk SMS and Bulk Email ? Email 2 SMS 2 Email - Developer Kit for Mobile Services integration - Voice Over IP services *From:* Hanh Le Bich [mailto:hanhmi...@gmail.com] *Sent:* 29 April 2014 12:25 PM *To:* Alexander Malysh *Cc:* hbil...@ecommunicate.biz; devel@kannel.org *Subject:* Re: Does opensmppbox and smsbox have serious memory issues? It's happy to get it now. Thank a lot. Regards, Tuan. On Tue, Apr 29, 2014 at 4:02 PM, Alexander Malysh <amal...@kannel.org> wrote: Hi, just checked daily snapshot, yes you can use it, this is uptodate. Alex Am 29.04.2014 um 04:44 schrieb Hanh Le Bich <hanhmi...@gmail.com>: Dear hbilman & development team, I'm willing to help to provide more evident but i have no background to work in IT fields. Cause my server has no internet connection thus i cannot get the lastest SVN trunk. Normally i download source files via the daily snapshot, is it ok? Regards, Hanh. On Mon, Apr 28, 2014 at 12:25 AM, <hbil...@ecommunicate.biz> wrote: Hi Kannel developers, Hanh posted his Valgrind research to the user group for smsbox and opensmppbox. His results seem interesting and so I'm copying them to this thread so the Kannel developers can view them. These results can be viewed by following the thread on Wed, Apr 23, 2014 at 3:41 AM, by Hanh Le Bich <hanhmi...@gmail.com> with the Subject: Re: 2 Questions re Redis/Debian. (The email subject is not related to this issue.) His research shows that opensmppbox and smsbox may have serious memory issues. I use the word "may" as until others have confirmed his results, there could be a mistake somewhere. Is there anyone who has a test environment that can follow his approach and confirm for the Kannel community if opensmppbox and smsbox have serious memory issues or if his testing approach has flaws? His approach is: Let me describe a little bit for my application back end. It's pretty simple: i make a loop that for each second, it push an sms via kannel CGI for 1K mobile numbers, that mean throughput is 1000 msg/sec. My kannel configuration is simple too, it's only smsbox -> bearerbox -> SMSC (via smpp), no file storage, no SQL, no dlr (actually dlr-mask=8). I even don't expect the sms can deliver > to all end users and the app run some hours per day only. That why i can play with the lasted SVN which don't care so much for the reliability. For smsbox: In the pass when using ver 1.4.3, it was fine for years. After upgrade to 1.5.0, after each few days, i realized smsbox is reset, then i found it exhaust my memory. It's funny that smsbox consume the mem and doesn't release. Example, if it occupies 50% your mem and you stop sms pushing, it will 50% forever except the box restarting. That's all, same server with no other tasks, same back end, just different kannel version. Just paste the valgrind sum in here: ==27581== LEAK SUMMARY: ==27581== definitely lost: 1,077,904 bytes in 67,369 blocks ==27581== indirectly lost: 673,660 bytes in 67,366 blocks ==27581== possibly lost: 160 bytes in 13 blocks ==27581== still reachable: 1,240 bytes in 39 blocks ==27581== suppressed: 0 bytes in 0 blocks ==27581== Reachable blocks (those to which a pointer was found) are not shown. ==27581== To see them, rerun with: --leak-check=full --show-leak-kinds=all ==27581== ==27581== For counts of detected and suppressed errors, rerun with: -v ==27581== ERROR SUMMARY: 3 errors from 3 contexts (suppressed: 45 from 10) For opensmppbox opensmppbox drains your memory 10 times faster than smsbox ==31087== Memcheck, a memory error detector ==31087== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al. ==31087== Using Valgrind-3.9.0 and LibVEX; rerun with -h for copyright info ==31087== Command: /usr/local/sbin/opensmppbox -v -d -- /etc/kannel/opensmppbox.conf ==31087== Parent PID: 31085 ==31087== ==31087== ==31087== HEAP SUMMARY: ==31087== in use at exit: 8,163,073 bytes in 36,550 blocks ==31087== total heap usage: 893,827 allocs, 857,277 frees, 295,662,079 bytes allocated ==31087== ==31087== 49 (32 direct, 17 indirect) bytes in 2 blocks are definitely lost in loss record 485 of 813 ==31087== at 0x4027434: malloc (vg_replace_malloc.c:291) ==31087== by 0x80970B3: gw_native_malloc (gwmem-native.c:87) ==31087== by 0x80A37A1: octstr_create_from_data_real (octstr.c:263) ==31087== by 0x80A3D81: octstr_duplicate_real (octstr.c:377) ==31087== by 0x808B106: cfg_get_real (cfg.c:670) ==31087== by 0x8052769: main (opensmppbox.c:2291) ==31087== ==31087== 79 (64 direct, 15 indirect) bytes in 4 blocks are definitely lost in loss record 529 of 813 ==31087== at 0x4027434: malloc (vg_replace_malloc.c:291) ==31087== by 0x80970B3: gw_native_malloc (gwmem-native.c:87) ==31087== by 0x80A37A1: octstr_create_from_data_real (octstr.c:263) ==31087== by 0x80A3D81: octstr_duplicate_real (octstr.c:377) ==31087== by 0x805D52F: msg_duplicate (msg-decl.h:80) ==31087== by 0x805651D: catenate_msg (opensmppbox.c:525) ==31087== by 0x805686B: check_multipart (opensmppbox.c:1481) ==31087== by 0x8057AD8: smpp_to_bearerbox (opensmppbox.c:1639) ==31087== by 0x80983AE: new_thread (gwthread-pthread.c:385) ==31087== by 0x46F9C38: start_thread (pthread_create.c:304) ==31087== by 0x482F78D: clone (clone.S:130) ==31087== ==31087== 100 (80 direct, 20 indirect) bytes in 5 blocks are definitely lost in loss record 577 of 813 ==31087== at 0x4027434: malloc (vg_replace_malloc.c:291) ==31087== by 0x80970B3: gw_native_malloc (gwmem-native.c:87) ==31087== by 0x80A37A1: octstr_create_from_data_real (octstr.c:263) ==31087== by 0x80A3D81: octstr_duplicate_real (octstr.c:377) ==31087== by 0x805D6FD: msg_duplicate (msg-decl.h:80) ==31087== by 0x805651D: catenate_msg (opensmppbox.c:525) ==31087== by 0x805686B: check_multipart (opensmppbox.c:1481) ==31087== by 0x8057AD8: smpp_to_bearerbox (opensmppbox.c:1639) ==31087== by 0x80983AE: new_thread (gwthread-pthread.c:385) ==31087== by 0x46F9C38: start_thread (pthread_create.c:304) ==31087== by 0x482F78D: clone (clone.S:130) ==31087== ==31087== 144 bytes in 1 blocks are possibly lost in loss record 610 of 813 ==31087== at 0x402574D: calloc (vg_replace_malloc.c:618) ==31087== by 0x40111FB: _dl_allocate_tls (dl-tls.c:300) ==31087== by 0x46FA5A0: pthread_create@@GLIBC_2.1 (allocatestack.c:580) ==31087== by 0x4412F2A: my_thread_global_init (in /usr/lib/i386-linux-gnu/libmysqlclient.so.18.0.0) ==31087== by 0x44112F7: my_init (in /usr/lib/i386-linux-gnu/libmysqlclient.so.18.0.0) ==31087== by 0x43EC93A: mysql_server_init (in /usr/lib/i386-linux-gnu/libmysqlclient.so.18.0.0) ==31087== by 0x43EE078: mysql_init (in /usr/lib/i386-linux-gnu/libmysqlclient.so.18.0.0) ==31087== by 0x8094F42: mysql_open_conn (dbpool_mysql.c:84) ==31087== by 0x80952C5: dbpool_increase (dbpool.c:194) ==31087== by 0x80953F6: dbpool_create (dbpool.c:160) ==31087== by 0x805B2B3: dlr_init_mysql (dlr_mysql.c:452) ==31087== by 0x8058AF3: dlr_init (dlr.c:254) ==31087== ==31087== 400 (256 direct, 144 indirect) bytes in 16 blocks are definitely lost in loss record 680 of 813 ==31087== at 0x4027434: malloc (vg_replace_malloc.c:291) ==31087== by 0x80970B3: gw_native_malloc (gwmem-native.c:87) ==31087== by 0x80A37A1: octstr_create_from_data_real (octstr.c:263) ==31087== by 0x80A3D81: octstr_duplicate_real (octstr.c:377) ==31087== by 0x805474D: run_smppbox (opensmppbox.c:2105) ==31087== by 0x80983AE: new_thread (gwthread-pthread.c:385) ==31087== by 0x46F9C38: start_thread (pthread_create.c:304) ==31087== by 0x482F78D: clone (clone.S:130) ==31087== ==31087== 2,160 bytes in 15 blocks are possibly lost in loss record 755 of 813 ==31087== at 0x402574D: calloc (vg_replace_malloc.c:618) ==31087== by 0x40111FB: _dl_allocate_tls (dl-tls.c:300) ==31087== by 0x46FA5A0: pthread_create@@GLIBC_2.1 (allocatestack.c:580) ==31087== by 0x8097F4F: gwthread_create_real (gwthread-pthread.c:475) ==31087== by 0x80547B3: run_smppbox (opensmppbox.c:2124) ==31087== by 0x80983AE: new_thread (gwthread-pthread.c:385) ==31087== by 0x46F9C38: start_thread (pthread_create.c:304) ==31087== by 0x482F78D: clone (clone.S:130) ==31087== ==31087== 2,160 bytes in 15 blocks are possibly lost in loss record 756 of 813 ==31087== at 0x402574D: calloc (vg_replace_malloc.c:618) ==31087== by 0x40111FB: _dl_allocate_tls (dl-tls.c:300) ==31087== by 0x46FA5A0: pthread_create@@GLIBC_2.1 (allocatestack.c:580) ==31087== by 0x8097F4F: gwthread_create_real (gwthread-pthread.c:475) ==31087== by 0x8052F97: main (opensmppbox.c:2156) ==31087== ==31087== 3,020 (1,040 direct, 1,980 indirect) bytes in 13 blocks are definitely lost in loss record 762 of 813 ==31087== at 0x4027434: malloc (vg_replace_malloc.c:291) ==31087== by 0x80970B3: gw_native_malloc (gwmem-native.c:87) ==31087== by 0x80A0DEF: gwlist_create_real (list.c:131) ==31087== by 0x805FC8C: sms_split (sms.c:339) ==31087== by 0x805517E: run_smppbox (opensmppbox.c:1008) ==31087== by 0x80983AE: new_thread (gwthread-pthread.c:385) ==31087== by 0x46F9C38: start_thread (pthread_create.c:304) ==31087== by 0x482F78D: clone (clone.S:130) ==31087== ==31087== 44,032 bytes in 43 blocks are possibly lost in loss record 798 of 813 ==31087== at 0x4027434: malloc (vg_replace_malloc.c:291) ==31087== by 0x40275BA: realloc (vg_replace_malloc.c:687) ==31087== by 0x809722B: gw_native_realloc (gwmem-native.c:115) ==31087== by 0x80A35EB: octstr_grow (octstr.c:192) ==31087== by 0x80A64B3: octstr_insert_data (octstr.c:1469) ==31087== by 0x80A672A: octstr_append_data (octstr.c:1499) ==31087== by 0x80A90F9: octstr_format_valist_real (octstr.c:2486) ==31087== by 0x80A9366: octstr_format (octstr.c:2469) ==31087== by 0x80534F5: boxc_route_msg_to_smsc (opensmppbox.c:1791) ==31087== by 0x8057AAE: smpp_to_bearerbox (opensmppbox.c:1638) ==31087== by 0x80983AE: new_thread (gwthread-pthread.c:385) ==31087== by 0x46F9C38: start_thread (pthread_create.c:304) ==31087== ==31087== 4,986,528 (77,472 direct, 4,909,056 indirect) bytes in 4,842 blocks are definitely lost in loss record 813 of 813 ==31087== at 0x4027434: malloc (vg_replace_malloc.c:291) ==31087== by 0x80970B3: gw_native_malloc (gwmem-native.c:87) ==31087== by 0x80A37A1: octstr_create_from_data_real (octstr.c:263) ==31087== by 0x80A3916: octstr_create_real (octstr.c:245) ==31087== by 0x80A908E: octstr_format_valist_real (octstr.c:2480) ==31087== by 0x80A9366: octstr_format (octstr.c:2469) ==31087== by 0x80534F5: boxc_route_msg_to_smsc (opensmppbox.c:1791) ==31087== by 0x8057AAE: smpp_to_bearerbox (opensmppbox.c:1638) ==31087== by 0x80983AE: new_thread (gwthread-pthread.c:385) ==31087== by 0x46F9C38: start_thread (pthread_create.c:304) ==31087== by 0x482F78D: clone (clone.S:130) ==31087== ==31087== LEAK SUMMARY: ==31087== definitely lost: 78,944 bytes in 4,882 blocks ==31087== indirectly lost: 4,911,232 bytes in 4,859 blocks ==31087== possibly lost: 48,496 bytes in 74 blocks ==31087== still reachable: 3,124,401 bytes in 26,735 blocks ==31087== suppressed: 0 bytes in 0 blocks ==31087== Reachable blocks (those to which a pointer was found) are not shown. ==31087== To see them, rerun with: --leak-check=full --show-leak-kinds=all ==31087== ==31087== For counts of detected and suppressed errors, rerun with: -v ==31087== ERROR SUMMARY: 10 errors from 10 contexts (suppressed: 45 from 10) Regards -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.kannel.org/pipermail/devel/attachments/20140502/d093ed3c/attachm ent.html> -------------- next part -------------- ==23650== Memcheck, a memory error detector ==23650== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al. ==23650== Using Valgrind-3.9.0 and LibVEX; rerun with -h for copyright info ==23650== Command: /usr/local/sbin/opensmppbox -v -d -- /etc/kannel/opensmppbox.conf ==23650== Parent PID: 23648 ==23650== ==23650== ==23650== HEAP SUMMARY: ==23650== in use at exit: 2,213,708 bytes in 25,777 blocks ==23650== total heap usage: 688,951 allocs, 663,174 frees, 227,552,203 bytes allocated ==23650== ==23650== 48 (32 direct, 16 indirect) bytes in 2 blocks are definitely lost in loss record 457 of 714 ==23650== at 0x4027434: malloc (vg_replace_malloc.c:291) ==23650== by 0x80978C3: gw_native_malloc (gwmem-native.c:87) ==23650== by 0x80A3FB1: octstr_create_from_data_real (octstr.c:263) ==23650== by 0x80A4591: octstr_duplicate_real (octstr.c:377) ==23650== by 0x8056DC6: smpp_to_bearerbox (opensmppbox.c:1569) ==23650== by 0x8098BBE: new_thread (gwthread-pthread.c:385) ==23650== by 0x46F9C38: start_thread (pthread_create.c:304) ==23650== by 0x482F78D: clone (clone.S:130) ==23650== ==23650== 49 (32 direct, 17 indirect) bytes in 2 blocks are definitely lost in loss record 458 of 714 ==23650== at 0x4027434: malloc (vg_replace_malloc.c:291) ==23650== by 0x80978C3: gw_native_malloc (gwmem-native.c:87) ==23650== by 0x80A3FB1: octstr_create_from_data_real (octstr.c:263) ==23650== by 0x80A4591: octstr_duplicate_real (octstr.c:377) ==23650== by 0x808B846: cfg_get_real (cfg.c:670) ==23650== by 0x8052839: main (opensmppbox.c:2292) ==23650== ==23650== 144 bytes in 1 blocks are possibly lost in loss record 519 of 714 ==23650== at 0x402574D: calloc (vg_replace_malloc.c:618) ==23650== by 0x40111FB: _dl_allocate_tls (dl-tls.c:300) ==23650== by 0x46FA5A0: pthread_create@@GLIBC_2.1 (allocatestack.c:580) ==23650== by 0x4412F2A: my_thread_global_init (in /usr/lib/i386-linux-gnu/libmysqlclient.so.18.0.0) ==23650== by 0x44112F7: my_init (in /usr/lib/i386-linux-gnu/libmysqlclient.so.18.0.0) ==23650== by 0x43EC93A: mysql_server_init (in /usr/lib/i386-linux-gnu/libmysqlclient.so.18.0.0) ==23650== by 0x43EE078: mysql_init (in /usr/lib/i386-linux-gnu/libmysqlclient.so.18.0.0) ==23650== by 0x8095752: mysql_open_conn (dbpool_mysql.c:84) ==23650== by 0x8095AD5: dbpool_increase (dbpool.c:200) ==23650== by 0x8095C06: dbpool_create (dbpool.c:166) ==23650== by 0x805B3D3: dlr_init_mysql (dlr_mysql.c:452) ==23650== by 0x8058BA3: dlr_init (dlr.c:257) ==23650== ==23650== 164 (80 direct, 84 indirect) bytes in 1 blocks are definitely lost in loss record 531 of 714 ==23650== at 0x4027434: malloc (vg_replace_malloc.c:291) ==23650== by 0x80978C3: gw_native_malloc (gwmem-native.c:87) ==23650== by 0x80A15FF: gwlist_create_real (list.c:131) ==23650== by 0x80600DC: sms_split (sms.c:339) ==23650== by 0x805520E: run_smppbox (opensmppbox.c:1008) ==23650== by 0x8098BBE: new_thread (gwthread-pthread.c:385) ==23650== by 0x46F9C38: start_thread (pthread_create.c:304) ==23650== by 0x482F78D: clone (clone.S:130) ==23650== ==23650== 450 (288 direct, 162 indirect) bytes in 18 blocks are definitely lost in loss record 589 of 714 ==23650== at 0x4027434: malloc (vg_replace_malloc.c:291) ==23650== by 0x80978C3: gw_native_malloc (gwmem-native.c:87) ==23650== by 0x80A3FB1: octstr_create_from_data_real (octstr.c:263) ==23650== by 0x80A4591: octstr_duplicate_real (octstr.c:377) ==23650== by 0x80547DD: run_smppbox (opensmppbox.c:2106) ==23650== by 0x8098BBE: new_thread (gwthread-pthread.c:385) ==23650== by 0x46F9C38: start_thread (pthread_create.c:304) ==23650== by 0x482F78D: clone (clone.S:130) ==23650== ==23650== 1,460 (1,184 direct, 276 indirect) bytes in 74 blocks are definitely lost in loss record 654 of 714 ==23650== at 0x4027434: malloc (vg_replace_malloc.c:291) ==23650== by 0x80978C3: gw_native_malloc (gwmem-native.c:87) ==23650== by 0x80A3FB1: octstr_create_from_data_real (octstr.c:263) ==23650== by 0x80A4591: octstr_duplicate_real (octstr.c:377) ==23650== by 0x805DB4D: msg_duplicate (msg-decl.h:80) ==23650== by 0x80565AD: catenate_msg (opensmppbox.c:525) ==23650== by 0x80568FB: check_multipart (opensmppbox.c:1481) ==23650== by 0x8057B70: smpp_to_bearerbox (opensmppbox.c:1639) ==23650== by 0x8098BBE: new_thread (gwthread-pthread.c:385) ==23650== by 0x46F9C38: start_thread (pthread_create.c:304) ==23650== by 0x482F78D: clone (clone.S:130) ==23650== ==23650== 1,555 (1,200 direct, 355 indirect) bytes in 75 blocks are definitely lost in loss record 656 of 714 ==23650== at 0x4027434: malloc (vg_replace_malloc.c:291) ==23650== by 0x80978C3: gw_native_malloc (gwmem-native.c:87) ==23650== by 0x80A3FB1: octstr_create_from_data_real (octstr.c:263) ==23650== by 0x80A4591: octstr_duplicate_real (octstr.c:377) ==23650== by 0x805D97F: msg_duplicate (msg-decl.h:80) ==23650== by 0x80565AD: catenate_msg (opensmppbox.c:525) ==23650== by 0x80568FB: check_multipart (opensmppbox.c:1481) ==23650== by 0x8057B70: smpp_to_bearerbox (opensmppbox.c:1639) ==23650== by 0x8098BBE: new_thread (gwthread-pthread.c:385) ==23650== by 0x46F9C38: start_thread (pthread_create.c:304) ==23650== by 0x482F78D: clone (clone.S:130) ==23650== ==23650== 2,160 bytes in 15 blocks are possibly lost in loss record 663 of 714 ==23650== at 0x402574D: calloc (vg_replace_malloc.c:618) ==23650== by 0x40111FB: _dl_allocate_tls (dl-tls.c:300) ==23650== by 0x46FA5A0: pthread_create@@GLIBC_2.1 (allocatestack.c:580) ==23650== by 0x809875F: gwthread_create_real (gwthread-pthread.c:475) ==23650== by 0x8053067: main (opensmppbox.c:2157) ==23650== ==23650== 2,304 bytes in 16 blocks are possibly lost in loss record 666 of 714 ==23650== at 0x402574D: calloc (vg_replace_malloc.c:618) ==23650== by 0x40111FB: _dl_allocate_tls (dl-tls.c:300) ==23650== by 0x46FA5A0: pthread_create@@GLIBC_2.1 (allocatestack.c:580) ==23650== by 0x809875F: gwthread_create_real (gwthread-pthread.c:475) ==23650== by 0x8054843: run_smppbox (opensmppbox.c:2125) ==23650== by 0x8098BBE: new_thread (gwthread-pthread.c:385) ==23650== by 0x46F9C38: start_thread (pthread_create.c:304) ==23650== by 0x482F78D: clone (clone.S:130) ==23650== ==23650== LEAK SUMMARY: ==23650== definitely lost: 2,816 bytes in 172 blocks ==23650== indirectly lost: 910 bytes in 165 blocks ==23650== possibly lost: 4,608 bytes in 32 blocks ==23650== still reachable: 2,205,374 bytes in 25,408 blocks ==23650== suppressed: 0 bytes in 0 blocks ==23650== Reachable blocks (those to which a pointer was found) are not shown. ==23650== To see them, rerun with: --leak-check=full --show-leak-kinds=all ==23650== ==23650== For counts of detected and suppressed errors, rerun with: -v ==23650== ERROR SUMMARY: 9 errors from 9 contexts (suppressed: 45 from 10)