Hi Alex, Any ideas on how to remove the memory issues for smsbox?
Rgds ---------------------------------------------------------------------- Date: Sat, 3 May 2014 10:06:54 +0700 From: Hanh Le Bich <hanhmi...@gmail.com> To: hbil...@ecommunicate.biz Cc: devel@kannel.org Subject: Re: Does opensmppbox and smsbox have serious memory issues? Message-ID: <CA+p4DGxn78_ErwKii=yrpgrouxe9bflmc8om757yju41up8...@mail.gmail.com> Content-Type: text/plain; charset="utf-8" Hello again, Here is valgrind report for the smsbox memory leaking. This is also the latest kannel revision 5089. For me, it still happen, i submit (broadcast) million SMS(s) per day within few hours at 1000 TPS of throughput. Thus each day, smsbox swallow ~ 400MB RAM and not release until server memory exhaust after few days. At this time, the box is auto restarted and i have my memory back, so on, again and again. I not sure, but in very low throughput system, i am not aware this issue. ==5730== LEAK SUMMARY: ==5730== definitely lost: 231,680 bytes in 14,480 blocks ==5730== indirectly lost: 144,790 bytes in 14,479 blocks ==5730== possibly lost: 0 bytes in 0 blocks ==5730== still reachable: 1,250 bytes in 40 blocks ==5730== suppressed: 0 bytes in 0 blocks ==5730== Reachable blocks (those to which a pointer was found) are not shown. ==5730== To see them, rerun with: --leak-check=full --show-leak-kinds=all ==5730== ==5730== For counts of detected and suppressed errors, rerun with: -v ==5730== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 45 from 10) Please find the report in the attached also this link: https://www.dropbox.com/s/yjrc6wmdm0rwonz/smsbox_memleakcheck_20140503.txt Regards, Hanh. On Fri, May 2, 2014 at 9:33 PM, <hbil...@ecommunicate.biz> wrote: > Hi Hand, > > > > Did you see the latest change by Rene committed to SVN ( > http://www.kannel.org/pipermail/devel/2014-April/005612.html) > > Does it fix the memory issue you reported? > > > > rgds > > > > > ---------------------------------------------------------------------- > ---------------------------------------------------------------------- > ---- > > Kind Regards > > > > Hillel Bilman > > Manager eCommunicate > > mailto: hbil...@ecommunicate.co.za > > Cell: 083-2300002 > > Landline: 011-443-6164 > > Fax: 088-011-443-6164 > > > > Social Media marketing campaigns (Facebook, Google+, Twitter, LinkedIn > etc) ? Interactive Websites - .mobi Sites ? Mobile Apps(Android, > iPhone, Blackberry, Nokia) - Premium Rated SMSs and short codes - SMS > competitions and campaigns ? Lead Generation - opt-in subscription > Billing ? MMS campaigns - USSD campaigns - WAP - Outlook SMS ? Bulk SMS and Bulk Email ? > Email 2 SMS 2 Email - Developer Kit for Mobile Services integration - > Voice Over IP services > > > > *From:* Hanh Le Bich [mailto:hanhmi...@gmail.com] > *Sent:* 29 April 2014 12:25 PM > *To:* Alexander Malysh > *Cc:* hbil...@ecommunicate.biz; devel@kannel.org > *Subject:* Re: Does opensmppbox and smsbox have serious memory issues? > > > > It's happy to get it now. Thank a lot. > > Regards, > Tuan. > > > > On Tue, Apr 29, 2014 at 4:02 PM, Alexander Malysh <amal...@kannel.org> > wrote: > > Hi, > > > > just checked daily snapshot, yes you can use it, this is uptodate. > > > > Alex > > > > Am 29.04.2014 um 04:44 schrieb Hanh Le Bich <hanhmi...@gmail.com>: > > > > Dear hbilman & development team, > > I'm willing to help to provide more evident but i have no background > to work in IT fields. > > Cause my server has no internet connection thus i cannot get the > lastest SVN trunk. Normally i download source files via the daily > snapshot, is it ok? > > Regards, Hanh. > > > > On Mon, Apr 28, 2014 at 12:25 AM, <hbil...@ecommunicate.biz> wrote: > > Hi Kannel developers, > > Hanh posted his Valgrind research to the user group for smsbox and > opensmppbox. His results seem interesting and so I'm copying them to > this thread so the Kannel developers can view them. > These results can be viewed by following the thread on Wed, Apr 23, > 2014 at > 3:41 AM, by Hanh Le Bich <hanhmi...@gmail.com> with the Subject: Re: 2 > Questions re Redis/Debian. (The email subject is not related to this > issue.) > > His research shows that opensmppbox and smsbox may have serious memory > issues. > I use the word "may" as until others have confirmed his results, there > could be a mistake somewhere. > Is there anyone who has a test environment that can follow his > approach and confirm for the Kannel community if opensmppbox and > smsbox have serious memory issues or if his testing approach has flaws? > > His approach is: > >> Let me describe a little bit for my application back end. It's > >> pretty > >> simple: i make a loop that for each second, it push an sms via > >> kannel CGI for 1K mobile numbers, that mean throughput is 1000 msg/sec. > >> My kannel configuration is simple too, it's only smsbox -> > >> bearerbox > >> -> SMSC (via smpp), no file storage, no SQL, no dlr (actually > dlr-mask=8). > > I even don't expect the sms can deliver > to all end users and the > > app > run some hours per day only. That why i > > can play with the lasted SVN which don't care so much for the > reliability. > > For smsbox: > >>In the pass when using ver 1.4.3, it was fine for years. After > >>upgrade to 1.5.0, after each few days, i realized smsbox is reset, > >>then i found it exhaust my memory. It's funny that smsbox consume > >>the mem and doesn't release. Example, if it occupies 50% your mem > >>and you stop sms pushing, it will 50% forever except the box restarting. > >> That's all, same server with no other tasks, same back end, just > >>different kannel version. > >> > >> Just paste the valgrind sum in here: > >> > >> ==27581== LEAK SUMMARY: > >> ==27581== definitely lost: 1,077,904 bytes in 67,369 blocks > >> ==27581== indirectly lost: 673,660 bytes in 67,366 blocks > >> ==27581== possibly lost: 160 bytes in 13 blocks > >> ==27581== still reachable: 1,240 bytes in 39 blocks > >> ==27581== suppressed: 0 bytes in 0 blocks > >> ==27581== Reachable blocks (those to which a pointer was found) are > >> not shown. > >> ==27581== To see them, rerun with: --leak-check=full > >> --show-leak-kinds=all ==27581== ==27581== For counts of detected > >> and suppressed errors, rerun with: -v ==27581== ERROR SUMMARY: 3 > >> errors from 3 contexts (suppressed: 45 from 10) > > For opensmppbox > >opensmppbox drains your memory 10 times faster than smsbox > ==31087== Memcheck, a memory error detector ==31087== Copyright (C) > 2002-2013, and GNU GPL'd, by Julian Seward et al. > ==31087== Using Valgrind-3.9.0 and LibVEX; rerun with -h for copyright > info ==31087== Command: /usr/local/sbin/opensmppbox -v -d -- > /etc/kannel/opensmppbox.conf ==31087== Parent PID: 31085 ==31087== > ==31087== ==31087== HEAP SUMMARY: > ==31087== in use at exit: 8,163,073 bytes in 36,550 blocks > ==31087== total heap usage: 893,827 allocs, 857,277 frees, 295,662,079 > bytes allocated > ==31087== > ==31087== 49 (32 direct, 17 indirect) bytes in 2 blocks are definitely > lost in loss record 485 of 813 > ==31087== at 0x4027434: malloc (vg_replace_malloc.c:291) > ==31087== by 0x80970B3: gw_native_malloc (gwmem-native.c:87) > ==31087== by 0x80A37A1: octstr_create_from_data_real (octstr.c:263) > ==31087== by 0x80A3D81: octstr_duplicate_real (octstr.c:377) > ==31087== by 0x808B106: cfg_get_real (cfg.c:670) > ==31087== by 0x8052769: main (opensmppbox.c:2291) > ==31087== > ==31087== 79 (64 direct, 15 indirect) bytes in 4 blocks are definitely > lost in loss record 529 of 813 > ==31087== at 0x4027434: malloc (vg_replace_malloc.c:291) > ==31087== by 0x80970B3: gw_native_malloc (gwmem-native.c:87) > ==31087== by 0x80A37A1: octstr_create_from_data_real (octstr.c:263) > ==31087== by 0x80A3D81: octstr_duplicate_real (octstr.c:377) > ==31087== by 0x805D52F: msg_duplicate (msg-decl.h:80) > ==31087== by 0x805651D: catenate_msg (opensmppbox.c:525) > ==31087== by 0x805686B: check_multipart (opensmppbox.c:1481) > ==31087== by 0x8057AD8: smpp_to_bearerbox (opensmppbox.c:1639) > ==31087== by 0x80983AE: new_thread (gwthread-pthread.c:385) > ==31087== by 0x46F9C38: start_thread (pthread_create.c:304) > ==31087== by 0x482F78D: clone (clone.S:130) > ==31087== > ==31087== 100 (80 direct, 20 indirect) bytes in 5 blocks are > definitely lost in loss record 577 of 813 > ==31087== at 0x4027434: malloc (vg_replace_malloc.c:291) > ==31087== by 0x80970B3: gw_native_malloc (gwmem-native.c:87) > ==31087== by 0x80A37A1: octstr_create_from_data_real (octstr.c:263) > ==31087== by 0x80A3D81: octstr_duplicate_real (octstr.c:377) > ==31087== by 0x805D6FD: msg_duplicate (msg-decl.h:80) > ==31087== by 0x805651D: catenate_msg (opensmppbox.c:525) > ==31087== by 0x805686B: check_multipart (opensmppbox.c:1481) > ==31087== by 0x8057AD8: smpp_to_bearerbox (opensmppbox.c:1639) > ==31087== by 0x80983AE: new_thread (gwthread-pthread.c:385) > ==31087== by 0x46F9C38: start_thread (pthread_create.c:304) > ==31087== by 0x482F78D: clone (clone.S:130) > ==31087== > ==31087== 144 bytes in 1 blocks are possibly lost in loss record 610 of 813 > ==31087== at 0x402574D: calloc (vg_replace_malloc.c:618) > ==31087== by 0x40111FB: _dl_allocate_tls (dl-tls.c:300) > ==31087== by 0x46FA5A0: pthread_create@@GLIBC_2.1 (allocatestack.c:580) > ==31087== by 0x4412F2A: my_thread_global_init (in > /usr/lib/i386-linux-gnu/libmysqlclient.so.18.0.0) > ==31087== by 0x44112F7: my_init (in > /usr/lib/i386-linux-gnu/libmysqlclient.so.18.0.0) > ==31087== by 0x43EC93A: mysql_server_init (in > /usr/lib/i386-linux-gnu/libmysqlclient.so.18.0.0) > ==31087== by 0x43EE078: mysql_init (in > /usr/lib/i386-linux-gnu/libmysqlclient.so.18.0.0) > ==31087== by 0x8094F42: mysql_open_conn (dbpool_mysql.c:84) > ==31087== by 0x80952C5: dbpool_increase (dbpool.c:194) > ==31087== by 0x80953F6: dbpool_create (dbpool.c:160) > ==31087== by 0x805B2B3: dlr_init_mysql (dlr_mysql.c:452) > ==31087== by 0x8058AF3: dlr_init (dlr.c:254) > ==31087== > ==31087== 400 (256 direct, 144 indirect) bytes in 16 blocks are > definitely lost in loss record 680 of 813 > ==31087== at 0x4027434: malloc (vg_replace_malloc.c:291) > ==31087== by 0x80970B3: gw_native_malloc (gwmem-native.c:87) > ==31087== by 0x80A37A1: octstr_create_from_data_real (octstr.c:263) > ==31087== by 0x80A3D81: octstr_duplicate_real (octstr.c:377) > ==31087== by 0x805474D: run_smppbox (opensmppbox.c:2105) > ==31087== by 0x80983AE: new_thread (gwthread-pthread.c:385) > ==31087== by 0x46F9C38: start_thread (pthread_create.c:304) > ==31087== by 0x482F78D: clone (clone.S:130) > ==31087== > ==31087== 2,160 bytes in 15 blocks are possibly lost in loss record > 755 of > 813 > ==31087== at 0x402574D: calloc (vg_replace_malloc.c:618) > ==31087== by 0x40111FB: _dl_allocate_tls (dl-tls.c:300) > ==31087== by 0x46FA5A0: pthread_create@@GLIBC_2.1 (allocatestack.c:580) > ==31087== by 0x8097F4F: gwthread_create_real (gwthread-pthread.c:475) > ==31087== by 0x80547B3: run_smppbox (opensmppbox.c:2124) > ==31087== by 0x80983AE: new_thread (gwthread-pthread.c:385) > ==31087== by 0x46F9C38: start_thread (pthread_create.c:304) > ==31087== by 0x482F78D: clone (clone.S:130) > ==31087== > ==31087== 2,160 bytes in 15 blocks are possibly lost in loss record > 756 of > 813 > ==31087== at 0x402574D: calloc (vg_replace_malloc.c:618) > ==31087== by 0x40111FB: _dl_allocate_tls (dl-tls.c:300) > ==31087== by 0x46FA5A0: pthread_create@@GLIBC_2.1 (allocatestack.c:580) > ==31087== by 0x8097F4F: gwthread_create_real (gwthread-pthread.c:475) > ==31087== by 0x8052F97: main (opensmppbox.c:2156) > ==31087== > ==31087== 3,020 (1,040 direct, 1,980 indirect) bytes in 13 blocks are > definitely lost in loss record 762 of 813 > ==31087== at 0x4027434: malloc (vg_replace_malloc.c:291) > ==31087== by 0x80970B3: gw_native_malloc (gwmem-native.c:87) > ==31087== by 0x80A0DEF: gwlist_create_real (list.c:131) > ==31087== by 0x805FC8C: sms_split (sms.c:339) > ==31087== by 0x805517E: run_smppbox (opensmppbox.c:1008) > ==31087== by 0x80983AE: new_thread (gwthread-pthread.c:385) > ==31087== by 0x46F9C38: start_thread (pthread_create.c:304) > ==31087== by 0x482F78D: clone (clone.S:130) > ==31087== > ==31087== 44,032 bytes in 43 blocks are possibly lost in loss record > 798 of > 813 > ==31087== at 0x4027434: malloc (vg_replace_malloc.c:291) > ==31087== by 0x40275BA: realloc (vg_replace_malloc.c:687) > ==31087== by 0x809722B: gw_native_realloc (gwmem-native.c:115) > ==31087== by 0x80A35EB: octstr_grow (octstr.c:192) > ==31087== by 0x80A64B3: octstr_insert_data (octstr.c:1469) > ==31087== by 0x80A672A: octstr_append_data (octstr.c:1499) > ==31087== by 0x80A90F9: octstr_format_valist_real (octstr.c:2486) > ==31087== by 0x80A9366: octstr_format (octstr.c:2469) > ==31087== by 0x80534F5: boxc_route_msg_to_smsc (opensmppbox.c:1791) > ==31087== by 0x8057AAE: smpp_to_bearerbox (opensmppbox.c:1638) > ==31087== by 0x80983AE: new_thread (gwthread-pthread.c:385) > ==31087== by 0x46F9C38: start_thread (pthread_create.c:304) > ==31087== > ==31087== 4,986,528 (77,472 direct, 4,909,056 indirect) bytes in 4,842 > blocks are definitely lost in loss record 813 of 813 > ==31087== at 0x4027434: malloc (vg_replace_malloc.c:291) > ==31087== by 0x80970B3: gw_native_malloc (gwmem-native.c:87) > ==31087== by 0x80A37A1: octstr_create_from_data_real (octstr.c:263) > ==31087== by 0x80A3916: octstr_create_real (octstr.c:245) > ==31087== by 0x80A908E: octstr_format_valist_real (octstr.c:2480) > ==31087== by 0x80A9366: octstr_format (octstr.c:2469) > ==31087== by 0x80534F5: boxc_route_msg_to_smsc (opensmppbox.c:1791) > ==31087== by 0x8057AAE: smpp_to_bearerbox (opensmppbox.c:1638) > ==31087== by 0x80983AE: new_thread (gwthread-pthread.c:385) > ==31087== by 0x46F9C38: start_thread (pthread_create.c:304) > ==31087== by 0x482F78D: clone (clone.S:130) > ==31087== > ==31087== LEAK SUMMARY: > ==31087== definitely lost: 78,944 bytes in 4,882 blocks > ==31087== indirectly lost: 4,911,232 bytes in 4,859 blocks > ==31087== possibly lost: 48,496 bytes in 74 blocks > ==31087== still reachable: 3,124,401 bytes in 26,735 blocks > ==31087== suppressed: 0 bytes in 0 blocks > ==31087== Reachable blocks (those to which a pointer was found) are > not shown. > ==31087== To see them, rerun with: --leak-check=full > --show-leak-kinds=all ==31087== ==31087== For counts of detected and > suppressed errors, rerun > with: -v ==31087== ERROR SUMMARY: 10 errors from 10 contexts (suppressed: > 45 > from 10) > > Regards > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://www.kannel.org/pipermail/devel/attachments/20140503/94ee18a2/attachm ent.html> -------------- next part -------------- ==5730== Memcheck, a memory error detector ==5730== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al. ==5730== Using Valgrind-3.9.0 and LibVEX; rerun with -h for copyright info ==5730== Command: /usr/local/sbin/smsbox -v -d -- /etc/kannel/kannel.conf ==5730== Parent PID: 5728 ==5730== ==5730== ==5730== HEAP SUMMARY: ==5730== in use at exit: 377,720 bytes in 28,999 blocks ==5730== total heap usage: 2,999,707 allocs, 2,970,708 frees, 399,074,706 bytes allocated ==5730== ==5730== 376,470 (231,680 direct, 144,790 indirect) bytes in 14,480 blocks are definitely lost in loss record 42 of 42 ==5730== at 0x4027434: malloc (vg_replace_malloc.c:291) ==5730== by 0x8081BD3: gw_native_malloc (gwmem-native.c:87) ==5730== by 0x808F091: octstr_create_from_data_real (octstr.c:263) ==5730== by 0x808F671: octstr_duplicate_real (octstr.c:377) ==5730== by 0x8056966: send_message (smsbox.c:344) ==5730== by 0x805C4D4: smsbox_req_handle (smsbox.c:2303) ==5730== by 0x805D95D: sendsms_thread (smsbox.c:2558) ==5730== by 0x8082C9E: new_thread (gwthread-pthread.c:385) ==5730== by 0x4415C38: start_thread (pthread_create.c:304) ==5730== by 0x482FD4D: clone (clone.S:130) ==5730== ==5730== LEAK SUMMARY: ==5730== definitely lost: 231,680 bytes in 14,480 blocks ==5730== indirectly lost: 144,790 bytes in 14,479 blocks ==5730== possibly lost: 0 bytes in 0 blocks ==5730== still reachable: 1,250 bytes in 40 blocks ==5730== suppressed: 0 bytes in 0 blocks ==5730== Reachable blocks (those to which a pointer was found) are not shown. ==5730== To see them, rerun with: --leak-check=full --show-leak-kinds=all ==5730== ==5730== For counts of detected and suppressed errors, rerun with: -v ==5730== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 45 from 10)