Hi I hope i m not bothering you,
first the kannel servers, we choose 8GB but its over estimated in our test we never reached 5 GB, and never go ferther of 150 httpd process even if we have set it in kannel side to reach 1000 httpd process. For the web app server 32 GB is litle over estimated too, in our test with fakesms we reached the 10 GB, and yes for tunning the web side we use apache bench (ab) the values i have given where choosed based on ab benchmarking ( setting up the 4500 max httpd process) as i said to not complicate more things i m quite sure that the web side is well done because we have worked years now and we know how to tune it up system and apache sides With fakesms we never reached the values we got with ab and has the web server working fine that's why the bottleneck could never be the web server We tuned even kannel apache side as i said in all the tests we never seen the kannel server reach one of his limits, RAM, CPU, nember of httpd process running simultanitly imagine a state that you have a kannel how can process 1000 process httpd but he is running just 100 with low memory and CPU usage, the web server / db server can handle 4500 httpd process and 4500 postgres process and they are both far from their capabilitys, in our tests with fakesms when sending the 30000 sms we reached the 4000 httpd process in web app but still as i said with apache bench (ab) test the servers handled quite fine 4500 ones so 4000 should be fine too so how could be after like 15000 sms the incoming sms drop radically to 1sms/5-6 seconds ????? PS : for the web side we could go ferver up with ab test and increasing the nembrer of httpd process like 6000 7000 and even further withaout running out of ressources but we dont see the use for it ----- Mail Original ----- De: "DHC Admin" <dhcad...@gmail.com> À: "Ahmed BOUDHRAA" <ahmed.boudh...@inbmi.edunet.tn> Cc: "spameden" <spame...@gmail.com>, users@kannel.org Envoyé: Lundi 9 Juin 2014 17:20:53 Objet: Re: Tunning up kannel Ahmed, I understand that each server has 8Gb, each Apache client, with the usual WEB server stuff is around 10Mb, each client. How fast is your process for each MO? Because I can't see how can a server with 8Gb hold up 2000 clients at the same time (assuming the process takes 0.5sec) Have you checked the "apachectl status" while sending the messages to check how busy it is? Have you tried using "ab" to stress the web server and be 100% sure it can handle that amount of traffic? Sorry I can not help you further on this, I don't know where is the problem, but I think it might not be a pure kannel thing, but a combination of the whole system. On Mon, Jun 9, 2014 at 12:55 PM, Ahmed BOUDHRAA < ahmed.boudh...@inbmi.edunet.tn > wrote: Yes yes this is what we think there will be anonther queue in smsbox, and yes we have enough resources as i said configuring the web at 4000 requests / s its about the 1/4 of his hardware capability and with that configuration we havent reached the 4000 request / s for the the MO Queue that i mentioned its in the png attached its taked from the web interface kannel status ----- Mail Original ----- De: "DHC Admin" < dhcad...@gmail.com > À: "Ahmed BOUDHRAA" < ahmed.boudh...@inbmi.edunet.tn > Cc: "spameden" < spame...@gmail.com >, users@kannel.org Envoyé: Lundi 9 Juin 2014 16:39:24 Objet: Re: Tunning up kannel Hi Ahmed Maybe the Bearebox has emptied the its queue and the queue moves to the smsbox? Are you sure you have enough resources to receive that much number of SMS at the same time? The Apache web server news to spawn (create) a big number of clients to handle the traffic, it might run out of resources for a few seconds and the smsbox could be resending the MOs later on, based on this configuration: http-request-retry integer If set, specifies how many retries should be performed for failing HTTP requests of sms-services. Defaults to 0, which means no retries should be performed and hence no HTTP request queuing is done. http-queue-delay integer If set, specifies how many seconds should pass within the HTTP queuing thread for retrying a failed HTTP request. Defaults to 10 sec. and is only obeyed if http-request-retry is set to a non-zero value. Have you tried to disable the HTTP retry and see if you loose any MO? Maybe they are getting retried. Please tell me where is that you see the MO Queue that you mention, are you just checking the status by console or using a HTML page? On Mon, Jun 9, 2014 at 11:51 AM, Ahmed BOUDHRAA < ahmed.boudh...@inbmi.edunet.tn > wrote: Thinx spameden Our plateform is made for publishing mainlly baccalaureate results, and tow other grades this is the main use right now, all those are about 240 000 canditates the fact is they may look like not much, but its a very important event that every parents/student attends the problematic its not about the nember but about the ammount of incoming/outgoing sms in a short time, last year we have reached the 800 sms /s in the operator side so you can imagine how it could be stressfull for us / the operators / the parents-students we cant allow any unavalability or too much wait, you can add to that that our plateform will do other things in the future :) for the throughput between as and the operators we fixied it in accordance with them every operator have specific capabilities but as i said our plateform can manage them at ease, what we want is to prepare our self, and to understand all the possible problem/solutions that we may have in using kannel. For the web server side, i cant assure that he is far from his real capabilities we are using RHEL web servers from long now. for the current configuration our web server can serve about 4000 requests / s i ll try below to describe a simple test we made usining fakesms: We are launching this test from our 3 kannel server simultanisouly : ./fakesmsc -i 0.001 -m 10000 "100 200 text test 000000*00000000" ==> this mean about 1000 sms /s from each kannel technically we could say that our portal should manage 3000 sms / s when we llaunch the test we can see on the kannel status in the box connections rubrique the "Queued (MO) started" 10000 x 3 and it take about 5 minutes to empty the three queue now what we want to understand is how the viewed queue is empty, the web server is far from his treatement capabilities and still messages are comming about 1-2 sms /s after about 15000 have reached the database. I dont know if i m clear normally when the queue is empty we have to have 30000 sms on the database side, wich mean there is another queue somewhere else this is our smskannel.conf : we are not using dlr nor internal storage: group = core admin-port = 13000 smsbox-port = 2776 admin-password = 123456 box-deny-ip = "*.*.*.*" box-allow-ip = "127.0.0.1" log-file = /var/log/kannel/kannel.log log-level = 0 access-log = /var/log/kannel/access_kannel.log access-log-clean = true access-log-format= SMS %t %l %i %p %P %b %F %I %k store-file =/var/log/kannel/sms.store dlr-storage = internal store-dump-freq = 5 sms-resend-freq = 60 sms-resend-retry = -1 #--------------------------------------------- # SMSC CONNECTIONS # # SMSC connections are created in bearerbox and they handle SMSC specific # protocol and message relying. You need these to actually receive and send # messages to handset, but can use GSM modems as virtual SMSCs group = smsc smsc = fake smsc-id = fake port = 10000 group = smsc smsc = smpp smsc-id = TT host = x.x.x.x port = xxxx log-file = /var/log/kannel/tt1.log log-level = 0 transceiver-mode = 1 receive-port = xxxx smsc-username = user smsc-password = pass system-type = 'VMA' interface-version = 34 preferred-smsc-id = TT group = smsc smsc = smpp smsc-id = TT host = x.x.x.x port = xxxx log-file = /var/log/kannel/tt2.log log-level = 0 transceiver-mode = 1 receive-port = xxxx smsc-username = user smsc-password = pass system-type = 'VMA' #service-type = 'test' interface-version = 34 preferred-smsc-id = TT group = smsc smsc = smpp smsc-id = TT host = x.x.x.x port = xxxx log-file = /var/log/kannel/tt3.log log-level = 0 transceiver-mode = 1 receive-port = xxxx smsc-username = user smsc-password = pass system-type = 'VMA' interface-version = 34 preferred-smsc-id = TT group = smsc smsc = smpp smsc-id = TT host = x.x.x.x port = xxxx log-file = /var/log/kannel/tt4.log log-level = 0 transceiver-mode = 1 receive-port = xxxx smsc-username = user smsc-password = pass system-type = 'VMA' interface-version = 34 preferred-smsc-id = TT group = smsc smsc = smpp smsc-id = TT host = x.x.x.x port = xxxx log-file = /var/log/kannel/tt5.log log-level = 0 transceiver-mode = 1 receive-port = xxxx smsc-username = user smsc-password = pass system-type = 'VMA' interface-version = 34 preferred-smsc-id = TT group = smsc smsc = smpp smsc-id = TT host = x.x.x.x port = xxxx log-file = /var/log/kannel/tt6.log log-level = 0 transceiver-mode = 1 receive-port = xxxx smsc-username = user smsc-password = pass system-type = 'VMA' interface-version = 34 preferred-smsc-id = TT group = smsc smsc = smpp smsc-id = TT host = x.x.x.x port = xxxx log-file = /var/log/kannel/tt7.log log-level = 0 transceiver-mode = 1 receive-port = xxxx smsc-username = user smsc-password = pass system-type = 'VMA' interface-version = 34 preferred-smsc-id = TT group = smsc smsc = smpp smsc-id = TT host = x.x.x.x port = xxxx log-file = /var/log/kannel/tt8.log log-level = 0 transceiver-mode = 1 receive-port = xxxx smsc-username = user smsc-password = pass system-type = 'VMA' interface-version = 34 preferred-smsc-id = TT group = smsc smsc = smpp smsc-id = TT host = x.x.x.x port = xxxx log-file = /var/log/kannel/tt9.log log-level = 0 transceiver-mode = 1 receive-port = xxxx smsc-username = user smsc-password = pass system-type = 'VMA' interface-version = 34 preferred-smsc-id = TT group = smsc smsc = smpp smsc-id = TT host = x.x.x.x port = xxxx log-file = /var/log/kannel/tt10.log log-level = 0 transceiver-mode = 1 receive-port = xxxx smsc-username = user smsc-password = pass system-type = 'VMA' interface-version = 34 preferred-smsc-id = TT group = smsc smsc = smpp smsc-id = TT host = x.x.x.x port = xxxx log-file = /var/log/kannel/tt11.log log-level = 0 transceiver-mode = 1 receive-port = xxxx smsc-username = user smsc-password = pass system-type = 'VMA' interface-version = 34 preferred-smsc-id = TT # This is a fake smsc connection, _only_ used to test the system and services. # It really cannot relay messages to actual handsets! #--------------------------------------------- # SMSBOX SETUP # # Smsbox(es) do higher-level SMS handling after they have been received from # SMS centers by bearerbox, or before they are given to bearerbox for delivery group = smsbox bearerbox-host = 127.0.0.1 sendsms-port = 8080 #global-sender = 13013 log-file = /var/log/kannel/smsbox.log log-level = 0 mo-recode = true reply-couldnotfetch = "----------------------------------" reply-couldnotrepresent = "-----------" reply-requestfailed = "-------------------------" reply-emptymessage = "*---------------------*" max-pending-requests = 1200 http-queue-delay = 2 http-request-retry = 100 #--------------------------------------------- # SEND-SMS USERS # # These users are used when Kannel smsbox sendsms interface is used to # send PUSH sms messages, i.e. calling URL like # http://kannel.machine:13013/cgi-bin/sendsms?username=tester&password=foobar. .. group = sendsms-user username = user password = pass user-allow-ip = ".e.e.e.e" forced-smsc = TT default-sender = 111111 max-messages = 4 concatenation = true #--------------------------------------------- # SERVICES # # These are 'responses' to sms PULL messages, i.e. messages arriving from # handsets. The response is based on message content. Only one sms-service is # applied, using the first one to match. group = sms-service keyword = default accept-x-kannel-headers = true catch-all = true max-messages =3 get-url = " http://site.site.tn/default.php?phone=%p&text=%a&smsc=%i&corp=%s&time=%t&smsid=%I&drv=%d&drs=%A&shortcode=%P " concatenation = true group = sms-service keyword = test accept-x-kannel-headers = true catch-all = true max-messages =0 get-url = " http://site.site.tn/test.php?phone=%p&text=%a&smsc=%i&corp=%s&time=%t&smsid=%I&drv=%d&drs=%A&shortcode=%P " concatenation = true group = sms-service keyword = bac accept-x-kannel-headers = true catch-all = true max-messages = 4 # it's better to put this parameter to 0 or you will have a lot Ack in your network get-url = " http://site.site.tn/default.php?phone=%p&text=%a&smsc=%i&corp=%s&time=%t&smsid=%I&drv=%d&drs=%A&shortcode=%P " concatenation = true group = sms-service keyword = neuf accept-x-kannel-headers = true #text = "No service specified" catch-all = true max-messages = 4 # it's better to put this parameter to 0 or you will have a lot Ack in your network get-url = " http://site.site.tn/default.php?phone=%p&text=%a&smsc=%i&corp=%s&time=%t&smsid=%I&drv=%d&drs=%A&shortcode=%P " concatenation = true group = sms-service keyword = six accept-x-kannel-headers = true #text = "No service specified" catch-all = true max-messages = 4 # it's better to put this parameter to 0 or you will have a lot Ack in your network get-url = " http://site.site.tn/default.php?phone=%p&text=%a&smsc=%i&corp=%s&time=%t&smsid=%I&drv=%d&drs=%A&shortcode=%P " concatenation = true group = sms-service keyword = ent accept-x-kannel-headers = true catch-all = true max-messages = 4 # it's better to put this parameter to 0 or you will have a lot Ack in your network get-url = " http://site.site.tn/entreceive.php?phone=%p&text=%a&smsc=%i&corp=%s&time=%t&smsid=%I&drv=%d&drs=%B&shortcode=%P " concatenation = true group = sms-service keyword = edu accept-x-kannel-headers = true catch-all = true max-messages = 5 # it's better to put this parameter to 0 or you will have a lot Ack in your network get-url = " http://site.site.tn/edureceive.php?phone=%p&text=%a&smsc=%i&corp=%s&time=%t&smsid=%I&drv=%d&drs=%B&shortcode=%P " concatenation = true thx for your patience ----- Mail Original ----- De: "spameden" < spame...@gmail.com > À: "Ahmed BOUDHRAA" < ahmed.boudh...@inbmi.edunet.tn > Cc: "DHC Admin" < dhcad...@gmail.com >, users@kannel.org Envoyé: Lundi 9 Juin 2014 14:40:50 Objet: Re: Tunning up kannel 2014-06-09 17:08 GMT+04:00 Ahmed BOUDHRAA < ahmed.boudh...@inbmi.edunet.tn > : HI thx for the relpay i was wondering if someone will help :) the fact is that we tunned the max-pending-requests and we ve concluded that about 800 is the best value and if we go up the results go worst, but the smsbox-max-pending we havent added such parametre; can you please explain it, is it, the smsbox-max-pending, is the queue between the smsbox and our portal? I can't imagine why do you need such high rate of sending MT messages for university unless you're spamming your students every second .. The speed also very much depends on your SMSC upstream providers, network link between you and them, TCP RTT (re-transmissions), etc. The general settings for throughput between you and smsc are: 1) throughput -- limits MT/sec 2) max-pending-submits -- unofficial parameter, controls how many "outstanding" operations between you and SMSC (e.g. unacknowledged submit_sm packets) Try tuning these two parameters to get maximum of your upstream SMSC. Do it carefully to not get THROTTLED errors or MAX_QUEUE errors. Consulting your provider is highly recommended. if i make it more simple: what is the difference between the tow variable that you gived to us : smsbox-max-pending and max-pending-requests ? and where are they placed, i mean if there is tow pending queue, i think one is in front of the bearerbox ( can i assume it is the max-pending-requests ?) and one between the bearerbox and the smsbox ? ( should it be the smsbox-max-pending ???) if my assumption are right i could be the bottleneck coz we havent touch such parametre, and if so the default value is about how much? check user-guide, it's available on the WEB: http://kannel.org/download/kannel-userguide-snapshot/userguide.html bottleneck is most likely your code in DLR-url parsing script not the kannel itself. try testing with different conditions: 1) without DLR reports at all 2) with DLR but without your url hit 3) with DLR and your URL Also, you might want to consider switching over to sqlbox, so you won't need to drag web-server each time report comes. thinx again for the help ----- Mail Original ----- De: "DHC Admin" < dhcad...@gmail.com > À: "Ahmed BOUDHRAA" < ahmed.boudh...@inbmi.edunet.tn > Cc: users@kannel.org Envoyé: Lundi 9 Juin 2014 13:50:46 Objet: Re: Tunning up kannel Hi Ahmed Have you tweaked the smsbox-max-pending and/or max-pending-requests? Check for them on the user guide On Wed, Jun 4, 2014 at 11:33 AM, Ahmed BOUDHRAA < ahmed.boudh...@inbmi.edunet.tn > wrote: Hi we are using kannel about 2 years in our institution and its woriking like a charm. we have high load traffic with 3 operators with 3 kannel gatways, our configuration is like this: - kannel 1 : Operator 1 : VM 2 x CPU 3 Ghz 8 Go Ram - kannel 2 : Operator 2 : VM 2 x CPU 3 Ghz 8 Go Ram - kannel 3 : Operator 3 : VM 2 x CPU 3 Ghz 8 Go Ram - Web Portal: for all kannel : VM 8 x CPU 3 Ghz 32 Go RAM - Postgres Database behind the web server : VM 8 x CPU 3 Ghz 32 Go RAM All those server are running under ESXi hypervisor in SAN environnement. Any way we are tunning all the plateform with several tools ( ApacheBench for web load and fakesms for kannel) we want to profit of all the hardware ressources with optimizing all componenents to reach the limits of the hardware, but still are still far behind the real capability of the hardware. The fact is we have reached 800 sms /s with the operators and its fine, but we want more not that we need it right now but just to master how every things work... The WEB/DB side tunning is "well" done due to fact that we have "mastered" it in our webs servers but the kannel side is litle different. with fakesms we are sending about 1000 sms / s from each kannel so we want to manage 3000 sms / s in our portal but all the kannel dosent send them with the speed we want even if they are not even litle solicited, we noticed that the bottle neck is between the smsbox and the bearerbox we dont know exactly but all 1000 sms arrive instantelly to the bearerbox in each kannel and a queue is formed between the bearerbox and smsbox, the smsbox send to the web server ( the portal ) about 1200 request / s wich is far from his limit ( we have tunning it to manage 3500 request / s ) I ve talked a lot but the main question is how we remove that so called queue smsbox/bearebox? to make the kannels send more than 1200 request/s so we can reach the limits in all the equippements all the system is running far behind his capabilitys. thinx for the help
