Re: troubleshooting MT queuing in Kannel / MT queue persistence question
2014-03-11 23:48 GMT+04:00 Jeff Thorn j...@thorntechnologies.com: Thanks spamden. 1 and 2 seem fine. We are working on 3 and 4. I don't know what would have caused the sudden change in throughput when everything was working fine. But I will send an update for the group if we figure it out. I am noticing something strange thoughour MT queues are starting to deplete now. However, if I look at the kannel store-status page, I still see messages queued from 4 hours ago. All new MT requests, however, are now going out immediately. Is this by design? Why doesn't kannel send queued messages before newer ones? Yes, it's by design, new submitted messages are going right after they are submitted unless you use an sqlbox with custom SQL query to get only certain MT messages to send to. What do you use for store? Spool or just a single file? Also check any errors in the dmesg or syslog. And check if kannel is swapping or anything else (free -m). Thanks, Jeff On Tue, Mar 11, 2014 at 3:05 PM, spameden spame...@gmail.com wrote: 2014-03-11 22:46 GMT+04:00 Jeff Thorn j...@thorntechnologies.com: Hi group, We currently have 7 different Tx binds setup to our SMSC. We have been sending MT volumes of up to 3,000,000 messages per day at a rate of almost 200 / second. We've been doing this for more than a year now with no problems. Suddenly about a week and a half ago, we noticed our throughput drop suddenly. The kannel status page shows we are only sending about 150 msgs / sec, but each of the 7 binds have over 50,000 MT messages queued. These queues eventually empty out, but some messages get delivered hours later than when they were sent by our platform. I am trying to troubleshoot this sudden drop in throughput and excessive queuing. The only thing I can think of is network latency between our network and the SMSC network. I don't know how to go about troubleshooting this though. Is there any other reason that might explain the sudden need for excessive queuing? There are many potential factors could be related to the actual sending speed. Also I don't find speed in the kannel's web interface very accurate. 1) set verbose mode to 0 for your SMSC logs and check if there are any throttling errors 2) check for maximum number of open files for kannel (ulimit under kannel user) and adjust if needed (this needed if you use kanel-store = spool) 3) check your backend speed (e.g. dlr-url script or MySQL db how much queries it can handle simultaneously). 4) contact your SMSC operator for troubleshooting (e.g. tcpdump the traffic and try to check if there are any network problems, massive tcp retransmits or anything else) Side question.if kannel was restarted with so many MT messages queued for delivery, would they all be lost? No, if you're using kannel-spool or kannel-store mode. Any assistance or tips for troubleshooting why so many messages are suddenly queuing would be extremely appreciated. Thanks, Jeff
RE: How to handle throttling and message queue full errors
Hello, it seems like for adjusting such behavior, it needs to be implemented, with a new directive (group smsc option).For requesting such thing from a developer, should I refer to de...@kannel.org mailing in order to find a developer who could work on a project like that? Thank you once again, From: vito...@hotmail.com To: users@kannel.org Subject: RE: How to handle throttling and message queue full errors Date: Mon, 10 Mar 2014 09:54:17 +0200 Hello again, Thanks for your responses. I have setup throughput and max-pending submits, but I need to setup this specific behavior on this smsc connection for these two errors, in case I receive them.I know that kannel by default has a stack of messages and the messages that receive an error goes down this stack. But I need to setup this specific behaviour.If receive a throttling error must stop sending anything for 60 seconds. Date: Fri, 7 Mar 2014 20:42:17 +0400 Subject: Re: How to handle throttling and message queue full errors From: spame...@gmail.com To: petrulis.val...@gmail.com CC: vito...@hotmail.com; users@kannel.org 2014-03-07 20:31 GMT+04:00 Valdas Petrulis petrulis.val...@gmail.com: You should use configarion variable 'throughput' under each of your SMSC: group = smsc smsc = smpp smsc-id = XXX ... ... throughput = 15 Kannel repeats messages from queue by default, if you did not modified configuration. Also, don't forget about max-pending-submits parameter. On Fri, Mar 7, 2014 at 5:33 PM, tolis man vito...@hotmail.com wrote: Hello, I would like to know if it's possible to handle some specific errors, like 1. 0x58 Throttling Error 2. 0x14 Message Queue Full -- There's a requirement saying that when I receive a throttling error message, the needed behaviour is: Stop sending any Submit_sm for 60 seconds. Any more outgoing Submit_sm's must be placed to the queue. The message, which get an error must be placed 1st in the queue. After 60 seconds timeout, continue sending normally, according queue. -- If I receive a Message Queue Full error, the needed behaviour is: Resend message. Try to resend it only 3-5 times. Every time you get an error, place message at very end of queue. -- Is it possible to adjust the above with configuration on the smsc group? Thank you in advance, -- Pagarbiai, Valdas Petrulis +37067310422 Skype: petrulis.valdas
Re: How to handle throttling and message queue full errors
Yeah, you can try devel mailing list as well as commerc...@kannel.org *commerc...@kannel.org commerc...@kannel.org* This mailing list allows users of Kannel to interact with commercial vendors, either letting aggregators offer SMSC connectivity here or any other commercial implications arround Kannel and SMS marketing. 2014-03-12 12:57 GMT+04:00 tolis man vito...@hotmail.com: Hello, it seems like for adjusting such behavior, it needs to be implemented, with a new directive (group smsc option). For requesting such thing from a developer, should I refer to de...@kannel.org mailing in order to find a developer who could work on a project like that? Thank you once again, -- From: vito...@hotmail.com To: users@kannel.org Subject: RE: How to handle throttling and message queue full errors Date: Mon, 10 Mar 2014 09:54:17 +0200 Hello again, Thanks for your responses. I have setup throughput and max-pending submits, but I need to setup this specific behavior on this smsc connection for these two errors, in case I receive them. I know that kannel by default has a stack of messages and the messages that receive an error goes down this stack. But I need to setup this specific behaviour. If receive a throttling error must stop sending anything for 60 seconds. -- Date: Fri, 7 Mar 2014 20:42:17 +0400 Subject: Re: How to handle throttling and message queue full errors From: spame...@gmail.com To: petrulis.val...@gmail.com CC: vito...@hotmail.com; users@kannel.org 2014-03-07 20:31 GMT+04:00 Valdas Petrulis petrulis.val...@gmail.com: You should use configarion variable 'throughput' under each of your SMSC: group = smsc smsc = smpp smsc-id = XXX ... ... throughput = 15 Kannel repeats messages from queue by default, if you did not modified configuration. Also, don't forget about max-pending-submits parameter. On Fri, Mar 7, 2014 at 5:33 PM, tolis man vito...@hotmail.com wrote: Hello, I would like to know if it's possible to handle some specific errors, like 1. 0x58 Throttling Error 2. 0x14 Message Queue Full -- There's a requirement saying that when I receive a throttling error message, the needed behaviour is: Stop sending any Submit_sm for 60 seconds. Any more outgoing Submit_sm's must be placed to the queue. The message, which get an error must be placed 1st in the queue. After 60 seconds timeout, continue sending normally, according queue. -- If I receive a Message Queue Full error, the needed behaviour is: Resend message. Try to resend it only 3-5 times. Every time you get an error, place message at very end of queue. -- Is it possible to adjust the above with configuration on the smsc group? Thank you in advance, -- Pagarbiai, *Valdas Petrulis* +37067310422 Skype: petrulis.valdas [image: Inline image 1]
Re: Same SMS, 2 Different MessageIDs
So, If in your DLR url you request %F ? Wouldn't that just do what you need? Provide the same ID back to you for all DLRs relating to one MT? Regards, Kyriacos On 11/03/2014 14:50, mydi...@gmail.com wrote: Hi Kyriacos, Yea sure, here is the screenshot of my db where dlrs are inserted: http://www.mydigia.com/images/ImG/2014-03-11_1248.png There are 2 records for the same number sent to, one sms sending. msgidk is the kannel %I (kannel internal messageID), which changes on each type, which is my problem... Thank you, Ali. On 11/03/2014 10:42, Kyriacos/Netsmart wrote: By the way, can you provide samples of these different IDs? Kyriacos On 10/03/2014 15:25, spameden wrote: 2014-03-10 17:15 GMT+04:00 mydi...@gmail.com mailto:mydi...@gmail.com mydi...@gmail.com mailto:mydi...@gmail.com: Thanks for your reply. As for the speed, you are right, it doesn't affect my kannel receiving/sending speed, but what happens is sending to 100numbes in the same URL, instead of sending to 100 number in 100urls (1number each url), affects the sending script speed. I send through send-sms cgi using php, and the speed is massively different between the two methods, which is why i need to use 100numbers in one url. Try tuning your script / database setup. Preferrably use nginx + php5-fpm over slow apache2. As for type=8, and dlr-mask. I completely have read the user guide, and know exactly how it works. My issue is different from that. And anyway I need type=8, for the reason you mentioned, I need to know the submission... Regardless of DLR-mask, shouldn't kannel provide ONLY 1 message ID for 1 sms send request? Why is it giving different message IDs for the same number, same request? Unless I am still missing a point... As I said earlier kannel's internal message scheme is suitable only for kannel's internal needs. Basically, each msgid inside the kannel (%I modifier) means unique identifier either for MT, MO or DLR message types. So, after you submit a batch of 100 numbers through send-sms cgi script with dlr_mask=31, kannel first returns DLR with status = 8 and its unique msgid and after sms get to the receiver you get a new DLR from your SMSC with status=1 and thus it has new msgid because it's a new DLR. You need to use your own msgid instead of %I and pass it to your dlr_url. I mean, regardless of type=1,..4,8,..., the messages id must be constant. That is the whole point, so the STATUS of one send request gets updated and the only way to identify which request, is using the messages id. No, the only constant messageid is SMSC message identifier, but you really shouldnt rely on it too, because if you use multiple SMSC they might be overlapping and different SMSC are using different types of message id (someone uses plain integers, someone hex, someone uses mixed text-integer type). Thank you, Ali. -- Kyriacos Sakkas Development Team Netsmart Tel: + 357 22 452565 Fax: + 357 22 452566 Email:kyria...@netsmart.com.cy http://www.netsmart.com.cy Taking Business to a New Level! ** Confidentiality Notice: The information contained in this email message may be privileged, confidential and protected from disclosure. If you are not the intended recipient, any dissemination, distribution, or copying of this email message is strictly prohibited. If you think that you have received this email message in error, please email the sender atkyria...@netsmart.com.cy ** -- Kyriacos Sakkas Development Team Netsmart Tel: + 357 22 452565 Fax: + 357 22 452566 Email: kyria...@netsmart.com.cy http://www.netsmart.com.cy Taking Business to a New Level! ** Confidentiality Notice: The information contained in this email message may be privileged, confidential and protected from disclosure. If you are not the intended recipient, any dissemination, distribution, or copying of this email message is strictly prohibited. If you think that you have received this email message in error, please email the sender at kyria...@netsmart.com.cy **
Re: Same SMS, 2 Different MessageIDs
Thanks for your reply Kyriacos. %F is SMSC provided. It could be delayed, it could be changing format (depending on each SMSC), it could indicate who my SMSC is, it could have different format. Like any standard provided, and normal SMSC, I must provide my own ID and be able to update status (type) based on that ID. You know what I mean? I can not request users to give their own messageID (not only technically is extra work and development for each of them, but no provider does that). What is everyone else using Kannel doing? I am just thrilled to know... Thank you, Ali. On 12/03/2014 09:15, Kyriacos/Netsmart wrote: So, If in your DLR url you request %F ? Wouldn't that just do what you need? Provide the same ID back to you for all DLRs relating to one MT? Regards, Kyriacos
Re: Same SMS, 2 Different MessageIDs
2014-03-12 14:31 GMT+04:00 mydi...@gmail.com mydi...@gmail.com:
Thanks for your reply Kyriacos.
%F is SMSC provided. It could be delayed, it could be changing format
(depending on each SMSC), it could indicate who my SMSC is, it could have
different format. Like any standard provided, and normal SMSC, I must
provide my own ID and be able to update status (type) based on that ID. You
know what I mean?
I can not request users to give their own messageID (not only technically
is extra work and development for each of them, but no provider does that).
What is everyone else using Kannel doing? I am just thrilled to know...
Again, you need to pass your OWN variable with identificator, e.g.:
dlr_url = url_encode('http://localhost/dlr.php?id=12312312status=%d
read this - http://www.kannel.org/pipermail/users/2010-July/011875.html
Thank you,
Ali.
On 12/03/2014 09:15, Kyriacos/Netsmart wrote:
So,
If in your DLR url you request %F ? Wouldn't that just do what you need?
Provide the same ID back to you for all DLRs relating to one MT?
Regards,
Kyriacos
Re: Redis delivery report DLR backend
Hi, I've activated the REDIS_DEBUG but the HMSET command works on redis-cli, but not with Kannel: bearerbox.log: 2014-03-12 15:54:14 [30938] [8] DEBUG: DLR[redis]: Adding DLR smsc=kannelfake, ts=8f3d43b4-f93c-4d8f-b681-35b658f74d90, src=51303, dst=39328000, mask=31, boxc= 2014-03-12 15:54:14 [30938] [8] DEBUG: Adding DLR into keystore 2014-03-12 15:54:14 [30938] [8] DEBUG: redis cmd: HMSET dlr:kannelfake:8f3d43b4-f93c-4d8f-b681-35b658f74d90:4677782 smsc kannelfake ts 8f3d43b4-f93c-4d8f-b681-35b658f74d90 source 51303 destination 39328 service user_mobyt_fake url http://my.fqdn.name/DlrManager.aspx31oa=%pda=%Psmsc=%idlrtype=%ddetail=%atempo=%Tid=%idelr=%Auser=%nforeign=%Fidgen=0|1|2|3|4|5|6mask _NULL_ boxc ? status 0 2014-03-12 15:54:14 [30938] [8] PANIC: /usr/sbin/bearerbox() [0x4a810c] 2014-03-12 15:54:14 [30938] [8] PANIC: /lib64/libpthread.so.0(+0xf710) [0x7f3d26ec0710] 2014-03-12 15:54:14 [30938] [8] PANIC: /usr/sbin/bearerbox() [0x48d431] 2014-03-12 15:54:14 [30938] [8] PANIC: /usr/sbin/bearerbox() [0x41fa63] 2014-03-12 15:54:14 [30938] [8] PANIC: /usr/sbin/bearerbox(dlr_add_real+0x3a3) [0x41cf63] 2014-03-12 15:54:14 [30938] [8] PANIC: /usr/sbin/bearerbox() [0x457b7f] 2014-03-12 15:54:14 [30938] [8] PANIC: /usr/sbin/bearerbox(smscconn_send+0x5f) [0x42406f] 2014-03-12 15:54:14 [30938] [8] PANIC: /usr/sbin/bearerbox(smsc2_rout+0x428) [0x417f68] 2014-03-12 15:54:14 [30938] [8] PANIC: /usr/sbin/bearerbox() [0x418e1d] 2014-03-12 15:54:14 [30938] [8] PANIC: /usr/sbin/bearerbox() [0x491da9] 2014-03-12 15:54:14 [30938] [8] PANIC: /lib64/libpthread.so.0(+0x79d1) [0x7f3d26eb89d1] 2014-03-12 15:54:14 [30938] [8] PANIC: /lib64/libc.so.6(clone+0x6d) [0x7f3d25c50b6d] Any clue for this? Thanks in advance Bagni 2014-03-06 12:37 GMT+01:00 Lorenzo Bagni ba...@networkweb.net: Hi all, I'm facing a blocking problem on redis backend for delivery report. I'm sendig correctly (and receive on my MT) but the bearerbox hangs when try to write on redis: smsc log: 2014-03-06 11:44:54 [25724] [7] DEBUG: SMPP[mobytusaprod]: Got PDU: 2014-03-06 11:44:54 [25724] [7] DEBUG: SMPP PDU 0x7f6c08000a10 dump: 2014-03-06 11:44:54 [25724] [7] DEBUG: type_name: submit_sm_resp 2014-03-06 11:44:54 [25724] [7] DEBUG: command_id: 2147483652 = 0x8004 2014-03-06 11:44:54 [25724] [7] DEBUG: command_status: 0 = 0x 2014-03-06 11:44:54 [25724] [7] DEBUG: sequence_number: 2 = 0x0002 2014-03-06 11:44:54 [25724] [7] DEBUG: message_id: 2014-03-06 11:44:54 [25724] [7] DEBUG:Octet string at 0x7f6c080011a0: 2014-03-06 11:44:54 [25724] [7] DEBUG: len: 36 2014-03-06 11:44:54 [25724] [7] DEBUG: size: 37 2014-03-06 11:44:54 [25724] [7] DEBUG: immutable: 0 2014-03-06 11:44:54 [25724] [7] DEBUG: data: 37 35 33 31 62 34 32 61 2d 66 34 34 66 2d 34 33 7531b42a-f44f-43 2014-03-06 11:44:54 [25724] [7] DEBUG: data: 38 31 2d 39 63 35 61 2d 36 31 37 65 32 32 33 31 81-9c5a-617e2231 2014-03-06 11:44:54 [25724] [7] DEBUG: data: 36 30 35 63 605c 2014-03-06 11:44:54 [25724] [7] DEBUG:Octet string dump ends. 2014-03-06 11:44:54 [25724] [7] DEBUG: SMPP PDU dump ends. 2014-03-06 11:44:54 [25724] [7] DEBUG: new group created `smpp' 2014-03-06 11:44:54 [25724] [7] DEBUG: group=`smpp' key=`mobyt_campaignid' value=`1245' 2014-03-06 11:44:54 [25724] [7] DEBUG: DLR[redis]: Adding DLR smsc=mobytusaprod, ts=7531b42a-f44f-4381-9c5a-617e2231605c, src=51303, dst=393284677782, mask=31, boxc= 2014-03-06 11:44:54 [25724] [7] DEBUG: Adding DLR into keystore bearerbox log: 2014-03-06 11:44:33 [25724] [9] DEBUG: Started thread 10 (gw/bb_boxc.c:boxc_sender) 2014-03-06 11:44:33 [25724] [10] DEBUG: Thread 10 (gw/bb_boxc.c:boxc_sender) maps to pid 25724. 2014-03-06 11:44:54 [25724] [9] DEBUG: boxc_receiver: sms received 2014-03-06 11:44:54 [25724] [9] DEBUG: send_msg: sending msg to box: 127.0.0.1 2014-03-06 11:44:54 [25724] [7] PANIC: /usr/sbin/bearerbox() [0x4a7f7c] 2014-03-06 11:44:54 [25724] [7] PANIC: /lib64/libpthread.so.0(+0xf710) [0x7f6c21df5710] 2014-03-06 11:44:54 [25724] [7] PANIC: /usr/sbin/bearerbox() [0x48d407] 2014-03-06 11:44:54 [25724] [7] PANIC: /usr/sbin/bearerbox() [0x41fa63] 2014-03-06 11:44:54 [25724] [7] PANIC: /usr/sbin/bearerbox(dlr_add_real+0x3a3) [0x41cf63] 2014-03-06 11:44:54 [25724] [7] PANIC: /usr/sbin/bearerbox() [0x4682c2] 2014-03-06 11:44:54 [25724] [7] PANIC: /usr/sbin/bearerbox() [0x469395] 2014-03-06 11:44:54 [25724] [7] PANIC: /usr/sbin/bearerbox() [0x491c19] 2014-03-06 11:44:54 [25724] [7] PANIC: /lib64/libpthread.so.0(+0x79d1) [0x7f6c21ded9d1] 2014-03-06 11:44:54 [25724] [7] PANIC: /lib64/libc.so.6(clone+0x6d) [0x7f6c20b85b6d] I've compiled kannel 1.5.0 trunk version 5081: Kannel bearerbox version `svn-runknown'. Build `Mar 5 2014 14:57:57', compiler `4.4.7 20120313 (Red Hat 4.4.7-4)'. System Linux, release 2.6.32-431.el6.x86_64, version #1 SMP Fri Nov 22 03:15:09 UTC 2013, machine x86_64.
