Re: [PATCH] Guaranteed throughput smsc-independent
Hi Donald, I think you hit the nail exactly on the head. AFAIK, there is a single thread at this point, sms_router, that is implementing bb_smscconn_sent. Therefore all sms in the outgoing smscs should be delayed. Kannel doesn't raise threads on demand. There may be an additional issue. This approach, which delays everything by a set amount if there is a throughput constraint, will utilize exactly half the available bandwidth. If for example an smsc has a throughput of 10 sms/1', and there is a queue of 20 sms, In the old way, assuming it works correctly, the first 10 should be send at once at time 0, and the rest at atime 1' with an average delay of 30/sms. With this approach, 1 sms would be sent every 6, with an average delay of 1'/sms. Coupled with the previous point, it could spell problems for queues in large installations that use many smscs to load balance. The advantage is that it offers solid throughput handling. I am a wap person, so I leave this decision for peoaple with a lot of smss. BR, Nikos - Original Message - From: Donald Jackson To: Damian Viano Cc: devel@kannel.org Sent: Wednesday, June 10, 2009 12:06 AM Subject: Re: [PATCH] Guaranteed throughput smsc-independent Ignore my last mail! I see these will be called per thread :) 2009/6/9 Donald Jackson donaldjs...@gmail.com I will review in more detail when I get a moment, my concern here is that you are now sleeping at the bearerbox layer which could effect delivery to other SMSC's. The current throughput limits although the logic is per-smsc, they sleep within their own threads as to not delay any others. Have you addressed this scenario? 2009/6/9 Damiαn Viano damian.vi...@buongiorno.com Hi list: I've seen kannel not respect throughput at all, at least with the fakesmsc and looking around find the following bug: http://redmine.kannel.org/issues/show/332 With the following patch attached: http://redmine.kannel.org/attachments/104/332-emi_patch_ack_v3.txt Inspired by that one, I've implemented a smsc-independent throughput patch. The idea is to enforce the throughput from the beaberbox side instead of having to implement the same login in every smsc. This is possible due to the bb_smscconn_sent() and bb_smscconn_send_failed() callbacks from the smscs implementation. They MUST call one of this callbacks after sending a message either successfully(_sent) or with failure(_failed), so we can make them sleep there, making sure they NEVER go over the configured throughput. There's only one downside to this smsc-independent approach which is that we don't, and can't (without cluttering the interface, AFAIK) know how much time the smsc takes in actually sending the sms, so we assume it takes nothing, this would, practically give us a somewhat smaller real throughput, but I though that's better than the previous behaviour (which for me, flooded the smsc). I've only tested this with fakesmsc so far, and only commented out the previous throughput implementation in that smsc, doing the rest is trivial and I can do it (or anyone else can), but this first iteration is to gather opinions about this approach. Once again the patch is against the current stable release, I can update it if needed, just let me know. Again I would love comments, questions, commits, rants, whatever :) For reference: diffstat kannel-1.4.3-throughput.patch bb_smscconn.c| 46 ++ smsc/smsc_fake.c |9 +++-- smscconn_p.h |2 ++ 3 files changed, 55 insertions(+), 2 deletions(-) Hope to help. Damiαn Viano(Des). P.D.: Also there's an info line in smsc_fake.c to count the number of sms, which should be removed from the final version, is only there for debugging purposes. -- Donald Jackson http://www.ddj.co.za/ donaldjster(a)gmail.com -- Donald Jackson http://www.ddj.co.za/ donaldjster(a)gmail.com
Re: Kannel not load balancing after restart and messages get stuck inqueue
Hi Alan, I must be the one thanking you for the discussion. You see I am a wap guy, and don't know of many of the issues that sms people face. The only concern is at startup. After it has bbox has started succesfully, and therefore sms_router, started as well, it doesn't matter. You can take all the smscs you want offline to service them. sms_router will work with the rest of them. The only issue is for startup. I cannot test the scenario you describe, because i don't have real smsc connections. However, if both connections are active the minute kannel starts (1st case0, there should be no problem. In the second case, the unavailable smsc should be commented out the configuration. I guess the only question is: Under what circumstances bbox will start, if the 2nd smsc is not responding? BR, Nikos - Original Message - From: Alan McNatty a...@catalyst.net.nz To: Nikos Balkanas nbalka...@gmail.com Cc: Alvaro Cornejo cornejo.alv...@gmail.com; devel@kannel.org Sent: Wednesday, June 10, 2009 6:14 AM Subject: Re: Kannel not load balancing after restart and messages get stuck inqueue Hi Nikos, OK, yeah I'm not about failure at start-up time. If an SMSC is off-line (SMPP timeout/retry during bind) kannel will start (simply config a bogus IP address for SMPP to test this out). An example is lets say we have 2 connections for load balancing. Now consider 1 smsc being taken offline (no tcp connection) as it is upgraded, we can still send through the other (this is also useful for DR - disaster recover, etc). So from my example above only 1 connection would be active the other in bind retry loop (wait X seconds, etc) - no tcp connection is established to smsc. If we re-started kannel it wouldn't be a good idea to wait for both links to become active before pushing messages out - this would mean waiting for the upgraded smsc to come back online, no? Again sorry if I'm talking at cross purposes here. Cheers, Alan Nikos Balkanas wrote: Hi Alan, If even 1 SMSc in configuration file fails to start, bearerbox will panic. There are several states to any SMSc, like connecting, going down, etc. Bearerbox expects all SMScs to be available on startup. The operative word is available. If a configured modem is not connected, that's a failure and you would have to take it off configuration to startup. After it starts, SMScs can go up or down. Isolated mode still needs SMScs for outgoing messages. The only mode that doesn't is Suspended. But i don't think that even this will startup without all SMScs available. Available is different than Active. Available means anything but Failed (Dead,etc.). An available SMSc could be down, or coming up or connecting. In this patch, router will wait to start until all SMScs become active. What is DR? BR, Nikos - Original Message - From: Alan McNatty a...@catalyst.net.nz To: Nikos Balkanas nbalka...@gmail.com Cc: Alvaro Cornejo cornejo.alv...@gmail.com; devel@kannel.org Sent: Wednesday, June 10, 2009 1:22 AM Subject: Re: Kannel not load balancing after restart and messages get stuck inqueue Hi Nikos, Sorry I haven't reviewed this whole thread in detail but was caught by .. bearerbox will not start unless all smscs are available. .. this worried me a bit. I'm concerned that in a DR scenario if 1 connection is down (and not coming up just yet) then I would like to be able to (re)start no problem (the other connections would start with this one remaining down .. trying to bind, etc). Is there not an alternative option in terms of (from init.d/startup perspective) start isolated, allowing some time (seconds) for the available connections to bind then setting the connections to active. Or have I missed the point here. Apologies if I'm muddying the water. I will try and follow more closely. Cheers, Alan Nikos Balkanas wrote: Dear Alvaro, I just managed to finish with my responsibilities, not on April 10, as I was hoping. Meanwhile, I haven't heard from you in a while, and I hope you are enjoying your vacations. Here is a patch to gw/bearerbox.c and gw/bb_smscconn.c, that doesn't start the sms_router, until all smscs (except the FAKE) are ACTIVE. If I am not mistaken, bearerbox will not start unless all smscs are available. So this is inline with this philosophy. I have tested it to the extend that it doesn't break anything, please let me know if it solves your problem. BR, Nikos - Original Message - From: Alvaro Cornejo cornejo.alv...@gmail.com To: Nikos Balkanas nbalka...@gmail.com Sent: Tuesday, March 31, 2009 4:56 PM Subject: Re: Kannel not load balancing after restart and messages get stuck inqueue Hi Nikos No problem. I undestand we are on a best effort support XD and this is not a critical issue. Thanks Alvaro |-| EnvΞ�β€�Ξ’Βe y Reciba Datos y mensajes de Texto (SMS) hacia y desde cualquier celular
Re: [PATCH] utils/mtabatch.c
Hi Nikos, -1 for this patch. 1) you don't want to send each part because bearerbox will split long message for you and ensure that it goes via the same SMSC 2) I don't see where mtbatch should core dump, please clarify. Thanks, Alex Am 09.06.2009 um 20:07 schrieb Nikos Balkanas: Hi, This is a simple patch to mtbatch, that if left as is should core dump. BR, Nikosmtbatch.diff
Re: [PATCH] Guaranteed throughput smsc-independent
Hi, not only this is issue. You want not sleep! Just think about it a bit. Hint: you want to process MOs and ACKs without delays... -1 Thanks, Alex P.S. I already posted patch that fixes this issue but not in ideal way. SMSC will not sleep but sending all allowed messages in the time slot at full speed instead of delaying each message. Am 10.06.2009 um 08:51 schrieb Nikos Balkanas: Hi Donald, I think you hit the nail exactly on the head. AFAIK, there is a single thread at this point, sms_router, that is implementing bb_smscconn_sent. Therefore all sms in the outgoing smscs should be delayed. Kannel doesn't raise threads on demand. There may be an additional issue. This approach, which delays everything by a set amount if there is a throughput constraint, will utilize exactly half the available bandwidth. If for example an smsc has a throughput of 10 sms/1', and there is a queue of 20 sms, In the old way, assuming it works correctly, the first 10 should be send at once at time 0, and the rest at atime 1' with an average delay of 30/sms. With this approach, 1 sms would be sent every 6, with an average delay of 1'/sms. Coupled with the previous point, it could spell problems for queues in large installations that use many smscs to load balance. The advantage is that it offers solid throughput handling. I am a wap person, so I leave this decision for peoaple with a lot of smss. BR, Nikos - Original Message - From: Donald Jackson To: Damián Viano Cc: devel@kannel.org Sent: Wednesday, June 10, 2009 12:06 AM Subject: Re: [PATCH] Guaranteed throughput smsc-independent Ignore my last mail! I see these will be called per thread :) 2009/6/9 Donald Jackson donaldjs...@gmail.com I will review in more detail when I get a moment, my concern here is that you are now sleeping at the bearerbox layer which could effect delivery to other SMSC's. The current throughput limits although the logic is per-smsc, they sleep within their own threads as to not delay any others. Have you addressed this scenario? 2009/6/9 Damiαn Viano damian.vi...@buongiorno.com Hi list: I've seen kannel not respect throughput at all, at least with the fakesmsc and looking around find the following bug: http://redmine.kannel.org/issues/show/332 With the following patch attached: http://redmine.kannel.org/attachments/104/332-emi_patch_ack_v3.txt Inspired by that one, I've implemented a smsc-independent throughput patch. The idea is to enforce the throughput from the beaberbox side instead of having to implement the same login in every smsc. This is possible due to the bb_smscconn_sent() and bb_smscconn_send_failed() callbacks from the smscs implementation. They MUST call one of this callbacks after sending a message either successfully(_sent) or with failure(_failed), so we can make them sleep there, making sure they NEVER go over the configured throughput. There's only one downside to this smsc-independent approach which is that we don't, and can't (without cluttering the interface, AFAIK) know how much time the smsc takes in actually sending the sms, so we assume it takes nothing, this would, practically give us a somewhat smaller real throughput, but I though that's better than the previous behaviour (which for me, flooded the smsc). I've only tested this with fakesmsc so far, and only commented out the previous throughput implementation in that smsc, doing the rest is trivial and I can do it (or anyone else can), but this first iteration is to gather opinions about this approach. Once again the patch is against the current stable release, I can update it if needed, just let me know. Again I would love comments, questions, commits, rants, whatever :) For reference: diffstat kannel-1.4.3-throughput.patch bb_smscconn.c| 46 ++ smsc/smsc_fake.c |9 +++-- smscconn_p.h |2 ++ 3 files changed, 55 insertions(+), 2 deletions(-) Hope to help. Damiαn Viano(Des). P.D.: Also there's an info line in smsc_fake.c to count the number of sms, which should be removed from the final version, is only there for debugging purposes. -- Donald Jackson http://www.ddj.co.za/ donaldjster(a)gmail.com -- Donald Jackson http://www.ddj.co.za/ donaldjster(a)gmail.com
DOC: Characters, Kannel and MT SMS - v.1.1 - comments are very appreciated
Dear all, since I've noticed that people has been asking a lot in the user@ list (and sometimes in the devel too) and that Kannel has performed a great improvement from 1.4.1 to 1.4.2 with the internal handling of UTF characters, I tough it was a good idea to write some notes on how to deal with foreign characters and kannel and examples for old and new users. The result is the document that I'm providing to the devel@ first for a first technical evaluation - any comment/note will be very appreciated. A PDF and .docx version is available at http://www.linkas.it/docs/kannel/ Both have still comments and notes available. The goal of this document is to be available to everyone asking as a starting reading point. I'm sure it can be improved, but with your help I'm sure we can make it better. Thanks for helping and thanks to those that through many posts in the lists wrote solutions and suggestions. Julien
Re: DOC: Characters, Kannel and MT SMS - v.1.1 - comments are very appreciated
Hi Julien, I think it would be great to have this on redmine wiki ? would you like to put it there? Thanks, Alex Am 10.06.2009 um 09:41 schrieb Julien Buratto: Dear all, since I've noticed that people has been asking a lot in the user@ list (and sometimes in the devel too) and that Kannel has performed a great improvement from 1.4.1 to 1.4.2 with the internal handling of UTF characters, I tough it was a good idea to write some notes on how to deal with foreign characters and kannel and examples for old and new users. The result is the document that I'm providing to the devel@ first for a first technical evaluation - any comment/note will be very appreciated. A PDF and .docx version is available at http://www.linkas.it/docs/kannel/ Both have still comments and notes available. The goal of this document is to be available to everyone asking as a starting reading point. I'm sure it can be improved, but with your help I'm sure we can make it better. Thanks for helping and thanks to those that through many posts in the lists wrote solutions and suggestions. Julien
Re: Kannel not load balancing after restart and messages get stuck inqueue
Hi Al, Please see inline answers. Kannel load balances all available smscs during operation. The only problem is that during startup, it starts sms_router too soon. From the discussion, i see that a patch may not even be necessary, it could be a configuration issue. Else I could always update the patch to account for more cases. BR, Nikos - Original Message - From: Alvaro Cornejo cornejo.alv...@gmail.com To: Alan McNatty a...@catalyst.net.nz Cc: Nikos Balkanas nbalka...@gmail.com; devel@kannel.org Sent: Wednesday, June 10, 2009 7:56 AM Subject: Re: Kannel not load balancing after restart and messages get stuck inqueue Hi Nikos I'm still around here... I've been busy this lasts days... I wish I was on vacations ;-) Regarding this patch, -I didn't had time to test it- but seems to solve the issue making kannel hold the queue assignment until all smsc's are online. However I do agree with Alan in the fact that that solution is not suitable since there can be many scenarios where an smsc will not come online or might be delayed at kannel startup and this will delay all message traffic. If one could describe me these scenarios (when bbox will startup succesfully, with smscs failing) I could implement them in the code. I'll be for a solution where Kannel recalculate its queue(loadbalance) for a given destination when a new destination for the given route/destination is available/unavailable and not only at startup. That's what it does all of the time. The only difference, at startup, it starts sms_router too soon. Another option might be to make kannel have an intermediate queue that is not assigned to an smsc until the smsc queue is less than 5min on the spscific smsc. This way in case of a restart, the 1st smsc to be available will get a moderate queue and the rest of smsc's will loadbalace the rest of messages once. Would something like that might be possible? I fear that an approach like this one will require a big remake of kannel queue/load management and I doubt kannel team will go for it. I can do that. No problem. But you don't need a patch for that, you can use configuration. Have you tried setting in smsc group: sms-outgoing-queue-limit = 50? This will push the first 50 sms to your smsc, and then push single sms, until your other smscs go online. Also I stick in the opinion that kannel shouldn't panic if it can't set active/enable an smsc on startup since there can be other smscs that can process messages for other routes/destinations. This issue is more important since the only workarroud is to edit kannel.conf comment the offending smsc and restart kannel and once the smsc is fixed, re-edit kannel.conf and restart kannel since so far, there is no way to make kannel reread the config file. We are in total agreement. I never liked this issue myself, either (both failing to startup and not reconfiguring on the fly). Once I test the patch I'll let you know. Probably this weekend. Please don't. Wait for the patch update. Regards and THANKS for your time Support. No pro, bro. Sorry for the delay, and for your ordeal. Mexico should be beautiful this time of the year. Alvaro |-| EnvΓe y Reciba Datos y mensajes de Texto (SMS) hacia y desde cualquier celular y Nextel en el PerΓΊ, MΓ©xico y en mas de 180 paises. Use aplicaciones 2 vias via SMS y GPRS online Visitenos en www.perusms.NET www.smsglobal.com.mx y www.pravcom.com On Tue, Jun 9, 2009 at 10:14 PM, Alan McNattya...@catalyst.net.nz wrote: Hi Nikos, OK, yeah I'm not about failure at start-up time. If an SMSC is off-line (SMPP timeout/retry during bind) kannel will start (simply config a bogus IP address for SMPP to test this out). An example is lets say we have 2 connections for load balancing. Now consider 1 smsc being taken offline (no tcp connection) as it is upgraded, we can still send through the other (this is also useful for DR - disaster recover, etc). So from my example above only 1 connection would be active the other in bind retry loop (wait X seconds, etc) - no tcp connection is established to smsc. If we re-started kannel it wouldn't be a good idea to wait for both links to become active before pushing messages out - this would mean waiting for the upgraded smsc to come back online, no? Again sorry if I'm talking at cross purposes here. Cheers, Alan Nikos Balkanas wrote: Hi Alan, If even 1 SMSc in configuration file fails to start, bearerbox will panic. There are several states to any SMSc, like connecting, going down, etc. Bearerbox expects all SMScs to be available on startup. The operative word is available. If a configured modem is not connected, that's a failure and you would have to take it off configuration to startup. After it starts, SMScs can go up or down. Isolated mode still needs SMScs for outgoing messages. The only
Re: [PATCH] utils/mtabatch.c
Hi, You are absolutely right. Please disregard. Yesterday I synced from cvs, and in send_message there was: Msg *part; which was never initialized, and then deliver_to_bearerbox(part) Sounds familiar? Now that I updated from cvs again, I can't see that code anymore. Seems you changed it on 30/4/2009. I don't know how I ended up yesterday with this outdated half baked copy. *Sigh* I guess it can happen to anyone. BR, Nikos - Original Message - From: Alexander Malysh To: Nikos Balkanas Cc: devel@kannel.org Devel Sent: Wednesday, June 10, 2009 10:24 AM Subject: Re: [PATCH] utils/mtabatch.c Hi Nikos, -1 for this patch. 1) you don't want to send each part because bearerbox will split long message for you and ensure that it goes via the same SMSC 2) I don't see where mtbatch should core dump, please clarify. Thanks, Alex Am 09.06.2009 um 20:07 schrieb Nikos Balkanas: Hi, This is a simple patch to mtbatch, that if left as is should core dump. BR, Nikosmtbatch.diff
Re: DOC: Characters, Kannel and MT SMS - v.1.1 - comments are very appreciated
Hi Alexander, wiki is a great idea for contributions to the doc. Is there a plan to move the whole kannel documentation to the wiki or it's just for contributions at first stage ? Julien PS: I cannot find the wiki on redmine.kannel.org, could you point me to it ? 2009/6/10 Alexander Malysh amal...@kannel.org: Hi Julien, I think it would be great to have this on redmine wiki ? would you like to put it there? Thanks, Alex Am 10.06.2009 um 09:41 schrieb Julien Buratto: Dear all, since I've noticed that people has been asking a lot in the user@ list (and sometimes in the devel too) and that Kannel has performed a great improvement from 1.4.1 to 1.4.2 with the internal handling of UTF characters, I tough it was a good idea to write some notes on how to deal with foreign characters and kannel and examples for old and new users. The result is the document that I'm providing to the devel@ first for a first technical evaluation - any comment/note will be very appreciated. A PDF and .docx version is available at http://www.linkas.it/docs/kannel/ Both have still comments and notes available. The goal of this document is to be available to everyone asking as a starting reading point. I'm sure it can be improved, but with your help I'm sure we can make it better. Thanks for helping and thanks to those that through many posts in the lists wrote solutions and suggestions. Julien -- Julien Buratto
Re: [PATCH] Guaranteed throughput smsc-independent
Actually, I don't think that even this patch can guarantee solid throughput. All I/O (including SMSc) is handled by a polling thread. AFAIK bb_smscconn_sent just puts the sms in the connection queue, where the connection layer takes over and sends it. If there is a queue for that SMSc, the connection layer will still try to empty it as fast as it can. IMHO, any robust throughput should be implemented in the connection layer. BR, Nikos - Original Message - From: Nikos Balkanas To: Donald Jackson ; Damian Viano Cc: devel@kannel.org Sent: Wednesday, June 10, 2009 9:51 AM Subject: Re: [PATCH] Guaranteed throughput smsc-independent Hi Donald, I think you hit the nail exactly on the head. AFAIK, there is a single thread at this point, sms_router, that is implementing bb_smscconn_sent. Therefore all sms in the outgoing smscs should be delayed. Kannel doesn't raise threads on demand. There may be an additional issue. This approach, which delays everything by a set amount if there is a throughput constraint, will utilize exactly half the available bandwidth. If for example an smsc has a throughput of 10 sms/1', and there is a queue of 20 sms, In the old way, assuming it works correctly, the first 10 should be send at once at time 0, and the rest at atime 1' with an average delay of 30/sms. With this approach, 1 sms would be sent every 6, with an average delay of 1'/sms. Coupled with the previous point, it could spell problems for queues in large installations that use many smscs to load balance. The advantage is that it offers solid throughput handling. I am a wap person, so I leave this decision for peoaple with a lot of smss. BR, Nikos - Original Message - From: Donald Jackson To: Damian Viano Cc: devel@kannel.org Sent: Wednesday, June 10, 2009 12:06 AM Subject: Re: [PATCH] Guaranteed throughput smsc-independent Ignore my last mail! I see these will be called per thread :) 2009/6/9 Donald Jackson donaldjs...@gmail.com I will review in more detail when I get a moment, my concern here is that you are now sleeping at the bearerbox layer which could effect delivery to other SMSC's. The current throughput limits although the logic is per-smsc, they sleep within their own threads as to not delay any others. Have you addressed this scenario? 2009/6/9 Damiαn Viano damian.vi...@buongiorno.com Hi list: I've seen kannel not respect throughput at all, at least with the fakesmsc and looking around find the following bug: http://redmine.kannel.org/issues/show/332 With the following patch attached: http://redmine.kannel.org/attachments/104/332-emi_patch_ack_v3.txt Inspired by that one, I've implemented a smsc-independent throughput patch. The idea is to enforce the throughput from the beaberbox side instead of having to implement the same login in every smsc. This is possible due to the bb_smscconn_sent() and bb_smscconn_send_failed() callbacks from the smscs implementation. They MUST call one of this callbacks after sending a message either successfully(_sent) or with failure(_failed), so we can make them sleep there, making sure they NEVER go over the configured throughput. There's only one downside to this smsc-independent approach which is that we don't, and can't (without cluttering the interface, AFAIK) know how much time the smsc takes in actually sending the sms, so we assume it takes nothing, this would, practically give us a somewhat smaller real throughput, but I though that's better than the previous behaviour (which for me, flooded the smsc). I've only tested this with fakesmsc so far, and only commented out the previous throughput implementation in that smsc, doing the rest is trivial and I can do it (or anyone else can), but this first iteration is to gather opinions about this approach. Once again the patch is against the current stable release, I can update it if needed, just let me know. Again I would love comments, questions, commits, rants, whatever :) For reference: diffstat kannel-1.4.3-throughput.patch bb_smscconn.c| 46 ++ smsc/smsc_fake.c |9 +++-- smscconn_p.h |2 ++ 3 files changed, 55 insertions(+), 2 deletions(-) Hope to help. Damiαn Viano(Des). P.D.: Also there's an info line in smsc_fake.c to count the number of sms, which should be removed from the final version, is only there for debugging purposes. -- Donald Jackson http://www.ddj.co.za/ donaldjster(a)gmail.com -- Donald Jackson http://www.ddj.co.za/
Re: DOC: Characters, Kannel and MT SMS - v.1.1 - comments are very appreciated
Hi, Great, but shouldn't this be in the traditional userguide format, i.e. xml, which can be converted with docbook to anything afterwards? I imagine it is not part of the userguide. BR, Nikos - Original Message - From: Alexander Malysh amal...@kannel.org To: Julien Buratto julien.bura...@gmail.com Cc: kannel_dev_mailinglist Devel devel@kannel.org Sent: Wednesday, June 10, 2009 10:49 AM Subject: Re: DOC: Characters,Kannel and MT SMS - v.1.1 - comments are very appreciated Hi Julien, I think it would be great to have this on redmine wiki ? would you like to put it there? Thanks, Alex Am 10.06.2009 um 09:41 schrieb Julien Buratto: Dear all, since I've noticed that people has been asking a lot in the user@ list (and sometimes in the devel too) and that Kannel has performed a great improvement from 1.4.1 to 1.4.2 with the internal handling of UTF characters, I tough it was a good idea to write some notes on how to deal with foreign characters and kannel and examples for old and new users. The result is the document that I'm providing to the devel@ first for a first technical evaluation - any comment/note will be very appreciated. A PDF and .docx version is available at http://www.linkas.it/docs/kannel/ Both have still comments and notes available. The goal of this document is to be available to everyone asking as a starting reading point. I'm sure it can be improved, but with your help I'm sure we can make it better. Thanks for helping and thanks to those that through many posts in the lists wrote solutions and suggestions. Julien
Re: Kannel not load balancing after restart and messages get stuck inqueue
On Wed, 2009-06-10 at 10:02, Nikos Balkanas wrote: I must be the one thanking you for the discussion. You see I am a wap guy, and don't know of many of the issues that sms people face. The only concern is at startup. After it has bbox has started succesfully, and therefore sms_router, started as well, it doesn't matter. You can take all the smscs you want offline to service them. sms_router will work with the rest of them. The only issue is for startup. I cannot test the scenario you describe, because i don't have real smsc connections. However, if both connections are active the minute kannel starts (1st case0, there should be no problem. In the second case, the unavailable smsc should be commented out the configuration. I guess the only question is: Under what circumstances bbox will start, if the 2nd smsc is not responding? Bearerbox should always start. Even if there is no one smsc active. Smsc can become active later for miscellaneous reasons. -- Kind regards, Milan
Re: [PATCH] custom MO Parameters for smsc-http
FYI, I've also commited the fix for kannel_receive_sms (removed the
dlrurl condition).
I've also added test/test_date and utils/decode_emimsg to .cvsignore.
Regards,
--
Alejandro Guerrieri
aguerri...@kannel.org
On 09/06/2009, at 19:05, Alejandro Guerrieri wrote:
Commited to CVS.
Regards,
--
Alejandro Guerrieri
aguerri...@kannel.org
On 09/06/2009, at 16:30, Alexander Malysh wrote:
now it's perfect ;)
would you like to commit or should I commit it?
Thanks,
Alex
Am 09.06.2009 um 16:06 schrieb Alejandro Guerrieri:
ups fixed :D
http://www.blogalex.com/wp-content/uploads/2009/06/kannel-http-mo-params2.patch
Regards,
--
Alejandro Guerrieri
aguerri...@kannel.org
On 09/06/2009, at 15:40, Alejandro Guerrieri wrote:
Oh, yes, sorry :D
I'd rather duplicate the msg...
--
Alejandro Guerrieri
aguerri...@kannel.org
On 09/06/2009, at 15:38, Alejandro Guerrieri wrote:
Yes, I have to move that afterwards, otherwise the
urltrans_fill_escape_codes call would panic:
2009-06-09 12:46:13 [15580] [6] PANIC: gwlib/octstr.c:2488:
seems_valid_real: Assertion `ostr-len = 0' failed. (Called
from gwlib/octstr.c:1552:octstr_split_words.)
...
2009-06-09 12:46:13 [15580] [6] PANIC: 0
bearerbox 0x0009136d gw_panic + 253
2009-06-09 12:46:13 [15580] [6] PANIC: 1
bearerbox 0x000925de log_thread_to + 750
2009-06-09 12:46:13 [15580] [6] PANIC: 2
bearerbox 0x0009b360
octstr_split_words + 48
2009-06-09 12:46:13 [15580] [6] PANIC: 3
bearerbox 0x000191e2
urltrans_fill_escape_codes + 114
2009-06-09 12:46:13 [15580] [6] PANIC: 4
bearerbox 0x0005d91b smsc_fake_create
+ 21803
Probably the bb_smscconn_receive call destroys the msg structure?
Regards,
--
Alejandro Guerrieri
aguerri...@kannel.org
On 09/06/2009, at 15:16, Alexander Malysh wrote:
only one last thing:
+msg-sms.account = octstr_duplicate(account);
+msg-sms.binfo = octstr_duplicate(binfo);
+if (ret == -1) {
+retmsg = octstr_create(Not accepted);
+retstatus = fm-status_error;
+} else {
+retmsg = urltrans_fill_escape_codes(fm-
message_sent, msg);
+retstatus = fm-status_sent;
+}
+ret = bb_smscconn_receive(conn, msg);
+}
are you sure you want check ret before bb_smscconn_receive ? ;)
Thanks,
Alex
Am 09.06.2009 um 13:23 schrieb Alejandro Guerrieri:
Ok, done!
http://www.blogalex.com/wp-content/uploads/2009/06/kannel-http-mo-params1.patch
I've fixed the dlr-url bug _only_ on the generic part. If no
objections, I'm doing a second patch to fix it on the kannel
smsc (since it's in fact a separate issue).
I've also fixed a small glitch on the dlrmsg response code (I
was using the error status code for successful submits as
well).
Last but not least, I've added url translation to the response
message, so now you can include escape codes on the response,
which may come handy on many cases (for example, to return
kannel's message id on the requests). Userguide part updated
accordingly.
Regards,
--
Alejandro Guerrieri
aguerri...@kannel.org
On 09/06/2009, at 11:27, Alexander Malysh wrote:
Am 09.06.2009 um 11:23 schrieb Alejandro Guerrieri:
Alex,
Regarding the charsets, I agree it's better to tackle it now
that leave it for later (aka, never ;) ).
I'm fixing the charset issue and the dlr bugs and resending
later.
thanks alex!
Regards,
--
Alejandro Guerrieri
aguerri...@kannel.org
On 09/06/2009, at 11:17, Alexander Malysh wrote:
Am 09.06.2009 um 11:05 schrieb Alejandro Guerrieri:
Alex,
Commenting inline below.
Regards,
--
Alejandro Guerrieri
aguerri...@kannel.org
On 09/06/2009, at 10:44, Alexander Malysh wrote:
Hi,
Seems you don't handle charset at all. Do you assume to
receive UTF-8 in any case? I don't think it's
applicable to generic interface. At least alt-charset
should be taken in account...
I'm not adding anything new here, the MO handling is built
on the original code for kannel_receive_sms (which was the
default interface for MO on the generic interface anyway).
There's no charset handling in there either. I agree it
makes sense to have charset handling, since an http
interface should be as flexible as possible, but is it a
showstopper at this point?
It is not really show stopper but if it will not be added
now it will remain so for ages. It is not a problem for
kannel_receive_sms because it's
well defined interface to smsbox which uses UTF-8.
I would prefer to see it implemented (there only few line
of code ;) )
some comments to patch below...
why is dlrurl required for DLR? We have dlrurl stored in
DLR storage...
+}
+else if (dlrurl != NULL dlrmask != 0 dlrmid !=
NULL) {
+/* we got a DLR, and we don't require additional
values */
+Msg *dlrmsg;
Idem... why is it
RE: [PATCH] custom MO Parameters for smsc-http
It's not so urgent but I could try, it could be interesting... I have 2 ideas so far... First idea == I could add 2 new config parameters to define 2 regex: - the first one, to find the start of the foreign id. - the second one, to find the end of the foreign id. For Claro, they could be defined like this: foreign-id-regex-begin = transaction-id foreign-id-regex-end = /transaction-id For Clickatell, they could be defined like this: foreign-id-regex-begin = ID: foreign-id-regex-end = For Brunet, they could be defined like this: foreign-id-regex-begin = MessageId= foreign-id-regex-end = In generic_parse_reply(), only in case of success, regex-begin will be used to find the beginning of the id. Starting from there, regex-end will be used to find the end of the id. If regex-end can't find a match, the id will be read until the end of text (body) and an ending CRLF will be removed. Then, the id will be used to call dlr_add() (like done in clickatell_parse_reply()). Second idea === It's basically the same idea except that only one regex will be used. For Claro, it could be defined like this: foreign-id-regex = transaction-id(.*)/transaction-id For Clickatell, it could be defined like this (if an id contains only digits): foreign-id-regex = ID: ([0-9]+) The offsets given by gw_regex_exec() in pmatch[1] could be used to extract the id. However, it could be more difficult to understand for people who are not regex experts... Any comment, opinion, suggestion or better idea? :) Franck From: Alexander Malysh Sent: Tuesday, June 09, 2009 6:05 PM Am 09.06.2009 um 17:55 schrieb Alejandro Guerrieri: Franck, The dlr part is inherited from the kannel smsc format. In the actual implementation there's no way to grab the remote message id from a request, and it'd make a lot of sense to be able to do it. I can try adding it, but don't refrain to do it so yourself if you feel the urge (you'll learn a lot about the code in the process!). yep, Alex is right here :) and we need more contributors :) Opinions? -- Alejandro Guerrieri aguerri...@kannel.org On 09/06/2009, at 17:47, Franck LAMASUTA wrote: Hi Alejandro , I work with Hervé who posted to this list the 25th of May (Claro http communication). It seems the patch you've done on the generic part could be very useful for us to manage our communications with Claro. It could be better than a specific implementation for Claro. However, I don't understand how your patch manages a DLR: In generic_receive_sms(), you call dlr_find() (line 1674 of the patched source) to retrieve a previous record stored in the DLR storage. It seems fine except that dlr_add() is never called in generic_parse_reply(), so I guess dlr_find() will always return NULL. Am I wrong??? In fact, when we submit a MT SMS to Claro with this request http://retail.mds.claro.com.br/MSE/api?profile=pwd=m ode=assync-deliveryANUM=BNUM=1234567TEXT=test we get a HTTP code 200 with this body: ?xml version=1.0 encoding=UTF-8 ? mse-response status-code0/status-code profileprofleID/profile transaction-id1020606201622668099/transaction-id /mse-response So I guess that the transaction-id should be used in generic_parse_reply() to call dlr_add(). Unfortunately, the current implementation does not allow to manage such an id, it only allows to search if the request was successful or not through the regex. If all my hypothesis are right... :-) I could try to patch generic_parse_reply() to manage also the id through another regex. It will be my first Kannel patch! ;-) If you prefer to do it, no problem, just let me know. If I'm wrong, please clarify how it works. Regards, Franck
Re: Kannel not load balancing after restart and messages get stuck inqueue
Milan P. Stanic wrote: On Wed, 2009-06-10 at 10:02, Nikos Balkanas wrote: I must be the one thanking you for the discussion. You see I am a wap guy, and don't know of many of the issues that sms people face. The only concern is at startup. After it has bbox has started succesfully, and therefore sms_router, started as well, it doesn't matter. You can take all the smscs you want offline to service them. sms_router will work with the rest of them. The only issue is for startup. I cannot test the scenario you describe, because i don't have real smsc connections. However, if both connections are active the minute kannel starts (1st case0, there should be no problem. In the second case, the unavailable smsc should be commented out the configuration. I guess the only question is: Under what circumstances bbox will start, if the 2nd smsc is not responding? Bearerbox should always start. Even if there is no one smsc active. Smsc can become active later for miscellaneous reasons. I agree 100% with Milan here. SMSC's are not in our control - there could be many points of failure between kannel and SMSC. Firewalls, VPN's etc may be unhappy and we lose connectivity .. we need kannel to be stable (in operation and restart, etc) and manage the reconnect and message distribution (SMPP is pretty good here). As Alvaro was suggesting earlier in the thread some sort of dynamic redistribution of queued message (maybe like keepalived or sim.) could be ideal .. but obviously some work .. Nikos I appreciate you have been busy doing some development. I can help test some of the SMSC 'stuff' (from SMPP) perspective - it's obviously in my interest here ;). As of now I haven't used a-lot of the routing features so this is a bit new to me (I will try and get back into it and hopefully make some suggestions). Cheers, Alan -- Alan McNatty, Wellington, New Zealand Catalyst IT Limited http://www.catalyst.net.nz/ DDI: +64 4 8032201
Re: Kannel not load balancing after restart and messages get stuck inqueue
Hi, I happen to agree also. But the fact is, that this is how kannel currently works. And i have to abide by that. Anyway, read my response to Alvaro, it may be just a configuration issue. BR, Nikos - Original Message - From: Alan McNatty a...@catalyst.net.nz To: nbalka...@gmail.com Cc: devel@kannel.org Sent: Wednesday, June 10, 2009 2:23 PM Subject: Re: Kannel not load balancing after restart and messages get stuck inqueue Milan P. Stanic wrote: On Wed, 2009-06-10 at 10:02, Nikos Balkanas wrote: I must be the one thanking you for the discussion. You see I am a wap guy, and don't know of many of the issues that sms people face. The only concern is at startup. After it has bbox has started succesfully, and therefore sms_router, started as well, it doesn't matter. You can take all the smscs you want offline to service them. sms_router will work with the rest of them. The only issue is for startup. I cannot test the scenario you describe, because i don't have real smsc connections. However, if both connections are active the minute kannel starts (1st case0, there should be no problem. In the second case, the unavailable smsc should be commented out the configuration. I guess the only question is: Under what circumstances bbox will start, if the 2nd smsc is not responding? Bearerbox should always start. Even if there is no one smsc active. Smsc can become active later for miscellaneous reasons. I agree 100% with Milan here. SMSC's are not in our control - there could be many points of failure between kannel and SMSC. Firewalls, VPN's etc may be unhappy and we lose connectivity .. we need kannel to be stable (in operation and restart, etc) and manage the reconnect and message distribution (SMPP is pretty good here). As Alvaro was suggesting earlier in the thread some sort of dynamic redistribution of queued message (maybe like keepalived or sim.) could be ideal .. but obviously some work .. Nikos I appreciate you have been busy doing some development. I can help test some of the SMSC 'stuff' (from SMPP) perspective - it's obviously in my interest here ;). As of now I haven't used a-lot of the routing features so this is a bit new to me (I will try and get back into it and hopefully make some suggestions). Cheers, Alan -- Alan McNatty, Wellington, New Zealand Catalyst IT Limited http://www.catalyst.net.nz/ DDI: +64 4 8032201
Re: Kannel not load balancing after restart and messages get stuck inqueue
On Wed, 2009-06-10 at 14:32, Nikos Balkanas wrote: I happen to agree also. But the fact is, that this is how kannel currently works. And i have to abide by that. Anyway, read my response to Alvaro, it may be just a configuration issue. I'm not sure if we are talking about same behavior of bearerbox but as I see it, bearerbox in 1.4.3 version (and some previous) starts and stays even if no one smsc (smsbox can't connect to SMSC) is running or in online status. If you talk about some specific configuration (smsc routing or load balancing) then you might be right. I didn't tested such use cases. Or, I misunderstood your post totally. - Original Message - From: Alan McNatty a...@catalyst.net.nz To: nbalka...@gmail.com Cc: devel@kannel.org Sent: Wednesday, June 10, 2009 2:23 PM Subject: Re: Kannel not load balancing after restart and messages get stuck inqueue Milan P. Stanic wrote: On Wed, 2009-06-10 at 10:02, Nikos Balkanas wrote: I must be the one thanking you for the discussion. You see I am a wap guy, and don't know of many of the issues that sms people face. The only concern is at startup. After it has bbox has started succesfully, and therefore sms_router, started as well, it doesn't matter. You can take all the smscs you want offline to service them. sms_router will work with the rest of them. The only issue is for startup. I cannot test the scenario you describe, because i don't have real smsc connections. However, if both connections are active the minute kannel starts (1st case0, there should be no problem. In the second case, the unavailable smsc should be commented out the configuration. I guess the only question is: Under what circumstances bbox will start, if the 2nd smsc is not responding? Bearerbox should always start. Even if there is no one smsc active. Smsc can become active later for miscellaneous reasons. I agree 100% with Milan here. SMSC's are not in our control - there could be many points of failure between kannel and SMSC. Firewalls, VPN's etc may be unhappy and we lose connectivity .. we need kannel to be stable (in operation and restart, etc) and manage the reconnect and message distribution (SMPP is pretty good here). As Alvaro was suggesting earlier in the thread some sort of dynamic redistribution of queued message (maybe like keepalived or sim.) could be ideal .. but obviously some work .. Nikos I appreciate you have been busy doing some development. I can help test some of the SMSC 'stuff' (from SMPP) perspective - it's obviously in my interest here ;). As of now I haven't used a-lot of the routing features so this is a bit new to me (I will try and get back into it and hopefully make some suggestions). -- Kind regards, Milan
Re: DOC: Characters, Kannel and MT SMS - v.1.1 - comments are very appreciated
Hi Julien, it's for the FAQ and additional docs in the first stage. Later we can look whether it make sense to port all userguige to wiki. Thanks, Alex Am 10.06.2009 um 10:17 schrieb Julien Buratto: Hi Alexander, wiki is a great idea for contributions to the doc. Is there a plan to move the whole kannel documentation to the wiki or it's just for contributions at first stage ? Julien PS: I cannot find the wiki on redmine.kannel.org, could you point me to it ? 2009/6/10 Alexander Malysh amal...@kannel.org: Hi Julien, I think it would be great to have this on redmine wiki ? would you like to put it there? Thanks, Alex Am 10.06.2009 um 09:41 schrieb Julien Buratto: Dear all, since I've noticed that people has been asking a lot in the user@ list (and sometimes in the devel too) and that Kannel has performed a great improvement from 1.4.1 to 1.4.2 with the internal handling of UTF characters, I tough it was a good idea to write some notes on how to deal with foreign characters and kannel and examples for old and new users. The result is the document that I'm providing to the devel@ first for a first technical evaluation - any comment/note will be very appreciated. A PDF and .docx version is available at http://www.linkas.it/docs/kannel/ Both have still comments and notes available. The goal of this document is to be available to everyone asking as a starting reading point. I'm sure it can be improved, but with your help I'm sure we can make it better. Thanks for helping and thanks to those that through many posts in the lists wrote solutions and suggestions. Julien -- Julien Buratto
Re: [PATCH] custom MO Parameters for smsc-http
Hi, I'm for the second idea, regex rocks ;) Thanks, Alex Am 10.06.2009 um 12:37 schrieb Franck LAMASUTA: It's not so urgent but I could try, it could be interesting... I have 2 ideas so far... First idea == I could add 2 new config parameters to define 2 regex: - the first one, to find the start of the foreign id. - the second one, to find the end of the foreign id. For Claro, they could be defined like this: foreign-id-regex-begin = transaction-id foreign-id-regex-end = /transaction-id For Clickatell, they could be defined like this: foreign-id-regex-begin = ID: foreign-id-regex-end = For Brunet, they could be defined like this: foreign-id-regex-begin = MessageId= foreign-id-regex-end = In generic_parse_reply(), only in case of success, regex-begin will be used to find the beginning of the id. Starting from there, regex- end will be used to find the end of the id. If regex-end can't find a match, the id will be read until the end of text (body) and an ending CRLF will be removed. Then, the id will be used to call dlr_add() (like done in clickatell_parse_reply()). Second idea === It's basically the same idea except that only one regex will be used. For Claro, it could be defined like this: foreign-id-regex = transaction-id(.*)/transaction-id For Clickatell, it could be defined like this (if an id contains only digits): foreign-id-regex = ID: ([0-9]+) The offsets given by gw_regex_exec() in pmatch[1] could be used to extract the id. However, it could be more difficult to understand for people who are not regex experts... Any comment, opinion, suggestion or better idea? :) Franck From: Alexander Malysh Sent: Tuesday, June 09, 2009 6:05 PM Am 09.06.2009 um 17:55 schrieb Alejandro Guerrieri: Franck, The dlr part is inherited from the kannel smsc format. In the actual implementation there's no way to grab the remote message id from a request, and it'd make a lot of sense to be able to do it. I can try adding it, but don't refrain to do it so yourself if you feel the urge (you'll learn a lot about the code in the process!). yep, Alex is right here :) and we need more contributors :) Opinions? -- Alejandro Guerrieri aguerri...@kannel.org On 09/06/2009, at 17:47, Franck LAMASUTA wrote: Hi Alejandro , I work with Hervé who posted to this list the 25th of May (Claro http communication). It seems the patch you've done on the generic part could be very useful for us to manage our communications with Claro. It could be better than a specific implementation for Claro. However, I don't understand how your patch manages a DLR: In generic_receive_sms(), you call dlr_find() (line 1674 of the patched source) to retrieve a previous record stored in the DLR storage. It seems fine except that dlr_add() is never called in generic_parse_reply(), so I guess dlr_find() will always return NULL. Am I wrong??? In fact, when we submit a MT SMS to Claro with this request http://retail.mds.claro.com.br/MSE/api?profile=pwd=m ode=assync-deliveryANUM=BNUM=1234567TEXT=test we get a HTTP code 200 with this body: ?xml version=1.0 encoding=UTF-8 ? mse-response status-code0/status-code profileprofleID/profile transaction-id1020606201622668099/transaction-id /mse-response So I guess that the transaction-id should be used in generic_parse_reply() to call dlr_add(). Unfortunately, the current implementation does not allow to manage such an id, it only allows to search if the request was successful or not through the regex. If all my hypothesis are right... :-) I could try to patch generic_parse_reply() to manage also the id through another regex. It will be my first Kannel patch! ;-) If you prefer to do it, no problem, just let me know. If I'm wrong, please clarify how it works. Regards, Franck
Re: Kannel not load balancing after restart and messages get stuckinqueue
Hi Milan,
Please edit gw/bb_smsccconn.c and go to smsc2_start:
for (i = 0; i gwlist_len(smsc_groups) .
(grp = gwlist_get(smsc_groups, i)) != NULL; i++) {
conn = smscconn_create(grp, 1);.
if (conn == NULL)
panic(0, Cannot start with SMSC connection failing);
gwlist_append(smsc_list, conn);
}
That's panic. That's what I am talking about.
I didn't write these rules, and l am sorry, I don't have the time to
continue this debate.
BR,
Nikos
- Original Message -
From: Milan P. Stanic m...@arvanta.net
To: devel@kannel.org
Sent: Wednesday, June 10, 2009 2:55 PM
Subject: Re: Kannel not load balancing after restart and messages get
stuckinqueue
On Wed, 2009-06-10 at 14:32, Nikos Balkanas wrote:
I happen to agree also. But the fact is, that this is how kannel
currently works. And i have to abide by that. Anyway, read my response to
Alvaro, it may be just a configuration issue.
I'm not sure if we are talking about same behavior of bearerbox but as
I see it, bearerbox in 1.4.3 version (and some previous) starts and
stays even if no one smsc (smsbox can't connect to SMSC) is running or in
online status.
If you talk about some specific configuration (smsc routing or load
balancing) then you might be right. I didn't tested such use cases.
Or, I misunderstood your post totally.
- Original Message - From: Alan McNatty a...@catalyst.net.nz
To: nbalka...@gmail.com
Cc: devel@kannel.org
Sent: Wednesday, June 10, 2009 2:23 PM
Subject: Re: Kannel not load balancing after restart and messages get
stuck inqueue
Milan P. Stanic wrote:
On Wed, 2009-06-10 at 10:02, Nikos Balkanas wrote:
I must be the one thanking you for the discussion. You see I am a wap
guy, and don't know of many of the issues that sms people face.
The only concern is at startup. After it has bbox has started
succesfully, and therefore sms_router, started as well, it doesn't
matter. You can take all the smscs you want offline to service them.
sms_router will work with the rest of them.
The only issue is for startup. I cannot test the scenario you
describe,
because i don't have real smsc connections. However, if both
connections
are active the minute kannel starts (1st case0, there should be no
problem. In the second case, the unavailable smsc should be
commented out
the configuration.
I guess the only question is: Under what circumstances bbox will
start,
if the 2nd smsc is not responding?
Bearerbox should always start. Even if there is no one smsc active.
Smsc can become active later for miscellaneous reasons.
I agree 100% with Milan here. SMSC's are not in our control - there
could be many points of failure between kannel and SMSC. Firewalls,
VPN's etc may be unhappy and we lose connectivity .. we need kannel to
be stable (in operation and restart, etc) and manage the reconnect and
message distribution (SMPP is pretty good here).
As Alvaro was suggesting earlier in the thread some sort of dynamic
redistribution of queued message (maybe like keepalived or sim.) could
be ideal .. but obviously some work ..
Nikos I appreciate you have been busy doing some development. I can help
test some of the SMSC 'stuff' (from SMPP) perspective - it's obviously
in my interest here ;). As of now I haven't used a-lot of the routing
features so this is a bit new to me (I will try and get back into it and
hopefully make some suggestions).
--
Kind regards, Milan
Re: [PATCH] Guaranteed throughput smsc-independent
here is a link to throttling branch: http://github.com/amalysh/kannel/tree/smpp-throttling only smpp implemented for now. Thanks, Alex Am 10.06.2009 um 10:20 schrieb Nikos Balkanas: Actually, I don't think that even this patch can guarantee solid throughput. All I/O (including SMSc) is handled by a polling thread. AFAIK bb_smscconn_sent just puts the sms in the connection queue, where the connection layer takes over and sends it. If there is a queue for that SMSc, the connection layer will still try to empty it as fast as it can. IMHO, any robust throughput should be implemented in the connection layer. BR, Nikos - Original Message - From: Nikos Balkanas To: Donald Jackson ; Damian Viano Cc: devel@kannel.org Sent: Wednesday, June 10, 2009 9:51 AM Subject: Re: [PATCH] Guaranteed throughput smsc-independent Hi Donald, I think you hit the nail exactly on the head. AFAIK, there is a single thread at this point, sms_router, that is implementing bb_smscconn_sent. Therefore all sms in the outgoing smscs should be delayed. Kannel doesn't raise threads on demand. There may be an additional issue. This approach, which delays everything by a set amount if there is a throughput constraint, will utilize exactly half the available bandwidth. If for example an smsc has a throughput of 10 sms/1', and there is a queue of 20 sms, In the old way, assuming it works correctly, the first 10 should be send at once at time 0, and the rest at atime 1' with an average delay of 30/sms. With this approach, 1 sms would be sent every 6, with an average delay of 1'/sms. Coupled with the previous point, it could spell problems for queues in large installations that use many smscs to load balance. The advantage is that it offers solid throughput handling. I am a wap person, so I leave this decision for peoaple with a lot of smss. BR, Nikos - Original Message - From: Donald Jackson To: Damián Viano Cc: devel@kannel.org Sent: Wednesday, June 10, 2009 12:06 AM Subject: Re: [PATCH] Guaranteed throughput smsc-independent Ignore my last mail! I see these will be called per thread :) 2009/6/9 Donald Jackson donaldjs...@gmail.com I will review in more detail when I get a moment, my concern here is that you are now sleeping at the bearerbox layer which could effect delivery to other SMSC's. The current throughput limits although the logic is per-smsc, they sleep within their own threads as to not delay any others. Have you addressed this scenario? 2009/6/9 Damiαn Viano damian.vi...@buongiorno.com Hi list: I've seen kannel not respect throughput at all, at least with the fakesmsc and looking around find the following bug: http://redmine.kannel.org/issues/show/332 With the following patch attached: http://redmine.kannel.org/attachments/104/332-emi_patch_ack_v3.txt Inspired by that one, I've implemented a smsc-independent throughput patch. The idea is to enforce the throughput from the beaberbox side instead of having to implement the same login in every smsc. This is possible due to the bb_smscconn_sent() and bb_smscconn_send_failed() callbacks from the smscs implementation. They MUST call one of this callbacks after sending a message either successfully(_sent) or with failure(_failed), so we can make them sleep there, making sure they NEVER go over the configured throughput. There's only one downside to this smsc-independent approach which is that we don't, and can't (without cluttering the interface, AFAIK) know how much time the smsc takes in actually sending the sms, so we assume it takes nothing, this would, practically give us a somewhat smaller real throughput, but I though that's better than the previous behaviour (which for me, flooded the smsc). I've only tested this with fakesmsc so far, and only commented out the previous throughput implementation in that smsc, doing the rest is trivial and I can do it (or anyone else can), but this first iteration is to gather opinions about this approach. Once again the patch is against the current stable release, I can update it if needed, just let me know. Again I would love comments, questions, commits, rants, whatever :) For reference: diffstat kannel-1.4.3-throughput.patch bb_smscconn.c| 46 ++ smsc/smsc_fake.c |9 +++-- smscconn_p.h |2 ++ 3 files changed, 55 insertions(+), 2 deletions(-) Hope to help. Damiαn Viano(Des). P.D.: Also there's an info line in smsc_fake.c to count the number of sms, which should be removed from the final version, is only there for debugging purposes. -- Donald Jackson http://www.ddj.co.za/ donaldjster(a)gmail.com -- Donald Jackson http://www.ddj.co.za/ donaldjster(a)gmail.com
Re: [PATCH] custom MO Parameters for smsc-http
Me too, second one is far more flexible and clean. -- Alejandro Guerrieri aguerri...@kannel.org On 10/06/2009, at 14:31, Alexander Malysh wrote: Hi, I'm for the second idea, regex rocks ;) Thanks, Alex Am 10.06.2009 um 12:37 schrieb Franck LAMASUTA: It's not so urgent but I could try, it could be interesting... I have 2 ideas so far... First idea == I could add 2 new config parameters to define 2 regex: - the first one, to find the start of the foreign id. - the second one, to find the end of the foreign id. For Claro, they could be defined like this: foreign-id-regex-begin = transaction-id foreign-id-regex-end = /transaction-id For Clickatell, they could be defined like this: foreign-id-regex-begin = ID: foreign-id-regex-end = For Brunet, they could be defined like this: foreign-id-regex-begin = MessageId= foreign-id-regex-end = In generic_parse_reply(), only in case of success, regex-begin will be used to find the beginning of the id. Starting from there, regex- end will be used to find the end of the id. If regex-end can't find a match, the id will be read until the end of text (body) and an ending CRLF will be removed. Then, the id will be used to call dlr_add() (like done in clickatell_parse_reply()). Second idea === It's basically the same idea except that only one regex will be used. For Claro, it could be defined like this: foreign-id-regex = transaction-id(.*)/transaction-id For Clickatell, it could be defined like this (if an id contains only digits): foreign-id-regex = ID: ([0-9]+) The offsets given by gw_regex_exec() in pmatch[1] could be used to extract the id. However, it could be more difficult to understand for people who are not regex experts... Any comment, opinion, suggestion or better idea? :) Franck From: Alexander Malysh Sent: Tuesday, June 09, 2009 6:05 PM Am 09.06.2009 um 17:55 schrieb Alejandro Guerrieri: Franck, The dlr part is inherited from the kannel smsc format. In the actual implementation there's no way to grab the remote message id from a request, and it'd make a lot of sense to be able to do it. I can try adding it, but don't refrain to do it so yourself if you feel the urge (you'll learn a lot about the code in the process!). yep, Alex is right here :) and we need more contributors :) Opinions? -- Alejandro Guerrieri aguerri...@kannel.org On 09/06/2009, at 17:47, Franck LAMASUTA wrote: Hi Alejandro , I work with Hervé who posted to this list the 25th of May (Claro http communication). It seems the patch you've done on the generic part could be very useful for us to manage our communications with Claro. It could be better than a specific implementation for Claro. However, I don't understand how your patch manages a DLR: In generic_receive_sms(), you call dlr_find() (line 1674 of the patched source) to retrieve a previous record stored in the DLR storage. It seems fine except that dlr_add() is never called in generic_parse_reply(), so I guess dlr_find() will always return NULL. Am I wrong??? In fact, when we submit a MT SMS to Claro with this request http://retail.mds.claro.com.br/MSE/api?profile=pwd=m ode=assync-deliveryANUM=BNUM=1234567TEXT=test we get a HTTP code 200 with this body: ?xml version=1.0 encoding=UTF-8 ? mse-response status-code0/status-code profileprofleID/profile transaction-id1020606201622668099/transaction-id /mse-response So I guess that the transaction-id should be used in generic_parse_reply() to call dlr_add(). Unfortunately, the current implementation does not allow to manage such an id, it only allows to search if the request was successful or not through the regex. If all my hypothesis are right... :-) I could try to patch generic_parse_reply() to manage also the id through another regex. It will be my first Kannel patch! ;-) If you prefer to do it, no problem, just let me know. If I'm wrong, please clarify how it works. Regards, Franck
Re: Kannel not load balancing after restart and messages get stuckinqueue
Hi Nikos,
this panic is reached only if SMSC module has wrong configuration.
This panic has nothing todo with active/offline/not connected SMSCs.
Thanks,
Alex
Am 10.06.2009 um 14:32 schrieb Nikos Balkanas:
Hi Milan,
Please edit gw/bb_smsccconn.c and go to smsc2_start:
for (i = 0; i gwlist_len(smsc_groups) .
(grp = gwlist_get(smsc_groups, i)) != NULL; i++) {
conn = smscconn_create(grp, 1);.
if (conn == NULL)
panic(0, Cannot start with SMSC connection failing);
gwlist_append(smsc_list, conn);
}
That's panic. That's what I am talking about.
I didn't write these rules, and l am sorry, I don't have the time to
continue this debate.
BR,
Nikos
- Original Message - From: Milan P. Stanic m...@arvanta.net
To: devel@kannel.org
Sent: Wednesday, June 10, 2009 2:55 PM
Subject: Re: Kannel not load balancing after restart and messages
get stuckinqueue
On Wed, 2009-06-10 at 14:32, Nikos Balkanas wrote:
I happen to agree also. But the fact is, that this is how kannel
currently works. And i have to abide by that. Anyway, read my
response to
Alvaro, it may be just a configuration issue.
I'm not sure if we are talking about same behavior of bearerbox but
as
I see it, bearerbox in 1.4.3 version (and some previous) starts and
stays even if no one smsc (smsbox can't connect to SMSC) is running
or in
online status.
If you talk about some specific configuration (smsc routing or load
balancing) then you might be right. I didn't tested such use cases.
Or, I misunderstood your post totally.
- Original Message - From: Alan McNatty a...@catalyst.net.nz
To: nbalka...@gmail.com
Cc: devel@kannel.org
Sent: Wednesday, June 10, 2009 2:23 PM
Subject: Re: Kannel not load balancing after restart and messages
get
stuck inqueue
Milan P. Stanic wrote:
On Wed, 2009-06-10 at 10:02, Nikos Balkanas wrote:
I must be the one thanking you for the discussion. You see I am
a wap
guy, and don't know of many of the issues that sms people face.
The only concern is at startup. After it has bbox has started
succesfully, and therefore sms_router, started as well, it
doesn't
matter. You can take all the smscs you want offline to service
them.
sms_router will work with the rest of them.
The only issue is for startup. I cannot test the scenario you
describe,
because i don't have real smsc connections. However, if both
connections
are active the minute kannel starts (1st case0, there should be
no
problem. In the second case, the unavailable smsc should be
commented out
the configuration.
I guess the only question is: Under what circumstances bbox
will start,
if the 2nd smsc is not responding?
Bearerbox should always start. Even if there is no one smsc
active.
Smsc can become active later for miscellaneous reasons.
I agree 100% with Milan here. SMSC's are not in our control - there
could be many points of failure between kannel and SMSC. Firewalls,
VPN's etc may be unhappy and we lose connectivity .. we need
kannel to
be stable (in operation and restart, etc) and manage the
reconnect and
message distribution (SMPP is pretty good here).
As Alvaro was suggesting earlier in the thread some sort of dynamic
redistribution of queued message (maybe like keepalived or sim.)
could
be ideal .. but obviously some work ..
Nikos I appreciate you have been busy doing some development. I
can help
test some of the SMSC 'stuff' (from SMPP) perspective - it's
obviously
in my interest here ;). As of now I haven't used a-lot of the
routing
features so this is a bit new to me (I will try and get back into
it and
hopefully make some suggestions).
--
Kind regards, Milan
Re: Kannel not load balancing after restart and messages get stuckinqueue
Alex
If so, why if I have a runing kannel with all smsc's up (modems) and
one of them died/get locked and restart kannel it will panic and the
config haven't changed ?
If do nothing I get a reconnecting status on kannel and all other
smsc's continue to work normally. I can even disconnect the modem and
replace it and kannel will reenable the smsc. Kannel will continue to
work properly. Also I can shutdown/up the smsc wih no problem.
The panic occurs if I do reset kannel with a locked or disconnected modem.
Regards
Alvaro
|-|
Envíe y Reciba Datos y mensajes de Texto (SMS) hacia y desde cualquier
celular y Nextel
en el Perú, México y en mas de 180 paises. Use aplicaciones 2 vias via
SMS y GPRS online
Visitenos en www.perusms.NET www.smsglobal.com.mx y
www.pravcom.com
On Wed, Jun 10, 2009 at 8:45 AM, Alexander Malyshamal...@kannel.org wrote:
Hi Nikos,
this panic is reached only if SMSC module has wrong configuration.
This panic has nothing todo with active/offline/not connected SMSCs.
Thanks,
Alex
Am 10.06.2009 um 14:32 schrieb Nikos Balkanas:
Hi Milan,
Please edit gw/bb_smsccconn.c and go to smsc2_start:
for (i = 0; i gwlist_len(smsc_groups) .
(grp = gwlist_get(smsc_groups, i)) != NULL; i++) {
conn = smscconn_create(grp, 1);.
if (conn == NULL)
panic(0, Cannot start with SMSC connection failing);
gwlist_append(smsc_list, conn);
}
That's panic. That's what I am talking about.
I didn't write these rules, and l am sorry, I don't have the time to
continue this debate.
BR,
Nikos
- Original Message - From: Milan P. Stanic m...@arvanta.net
To: devel@kannel.org
Sent: Wednesday, June 10, 2009 2:55 PM
Subject: Re: Kannel not load balancing after restart and messages get
stuckinqueue
On Wed, 2009-06-10 at 14:32, Nikos Balkanas wrote:
I happen to agree also. But the fact is, that this is how kannel
currently works. And i have to abide by that. Anyway, read my response
to
Alvaro, it may be just a configuration issue.
I'm not sure if we are talking about same behavior of bearerbox but as
I see it, bearerbox in 1.4.3 version (and some previous) starts and
stays even if no one smsc (smsbox can't connect to SMSC) is running or in
online status.
If you talk about some specific configuration (smsc routing or load
balancing) then you might be right. I didn't tested such use cases.
Or, I misunderstood your post totally.
- Original Message - From: Alan McNatty a...@catalyst.net.nz
To: nbalka...@gmail.com
Cc: devel@kannel.org
Sent: Wednesday, June 10, 2009 2:23 PM
Subject: Re: Kannel not load balancing after restart and messages get
stuck inqueue
Milan P. Stanic wrote:
On Wed, 2009-06-10 at 10:02, Nikos Balkanas wrote:
I must be the one thanking you for the discussion. You see I am a wap
guy, and don't know of many of the issues that sms people face.
The only concern is at startup. After it has bbox has started
succesfully, and therefore sms_router, started as well, it doesn't
matter. You can take all the smscs you want offline to service them.
sms_router will work with the rest of them.
The only issue is for startup. I cannot test the scenario you
describe,
because i don't have real smsc connections. However, if both
connections
are active the minute kannel starts (1st case0, there should be no
problem. In the second case, the unavailable smsc should be
commented out
the configuration.
I guess the only question is: Under what circumstances bbox will
start,
if the 2nd smsc is not responding?
Bearerbox should always start. Even if there is no one smsc active.
Smsc can become active later for miscellaneous reasons.
I agree 100% with Milan here. SMSC's are not in our control - there
could be many points of failure between kannel and SMSC. Firewalls,
VPN's etc may be unhappy and we lose connectivity .. we need kannel to
be stable (in operation and restart, etc) and manage the reconnect and
message distribution (SMPP is pretty good here).
As Alvaro was suggesting earlier in the thread some sort of dynamic
redistribution of queued message (maybe like keepalived or sim.) could
be ideal .. but obviously some work ..
Nikos I appreciate you have been busy doing some development. I can
help
test some of the SMSC 'stuff' (from SMPP) perspective - it's obviously
in my interest here ;). As of now I haven't used a-lot of the routing
features so this is a bit new to me (I will try and get back into it
and
hopefully make some suggestions).
--
Kind regards, Milan
Re: Kannel not load balancing after restart and messages get stuck inqueue
Hi Nikos Thanks for your answers I'll try sms-outgoing-queue-limit parameter and see what happens Regards Alvaro |-| Envíe y Reciba Datos y mensajes de Texto (SMS) hacia y desde cualquier celular y Nextel en el Perú, México y en mas de 180 paises. Use aplicaciones 2 vias via SMS y GPRS online Visitenos en www.perusms.NET www.smsglobal.com.mx y www.pravcom.com 2009/6/10 Nikos Balkanas nbalka...@gmail.com: Hi Al, Please see inline answers. Kannel load balances all available smscs during operation. The only problem is that during startup, it starts sms_router too soon. From the discussion, i see that a patch may not even be necessary, it could be a configuration issue. Else I could always update the patch to account for more cases. BR, Nikos - Original Message - From: Alvaro Cornejo cornejo.alv...@gmail.com To: Alan McNatty a...@catalyst.net.nz Cc: Nikos Balkanas nbalka...@gmail.com; devel@kannel.org Sent: Wednesday, June 10, 2009 7:56 AM Subject: Re: Kannel not load balancing after restart and messages get stuck inqueue Hi Nikos I'm still around here... I've been busy this lasts days... I wish I was on vacations ;-) Regarding this patch, -I didn't had time to test it- but seems to solve the issue making kannel hold the queue assignment until all smsc's are online. However I do agree with Alan in the fact that that solution is not suitable since there can be many scenarios where an smsc will not come online or might be delayed at kannel startup and this will delay all message traffic. If one could describe me these scenarios (when bbox will startup succesfully, with smscs failing) I could implement them in the code. I'll be for a solution where Kannel recalculate its queue(loadbalance) for a given destination when a new destination for the given route/destination is available/unavailable and not only at startup. That's what it does all of the time. The only difference, at startup, it starts sms_router too soon. Another option might be to make kannel have an intermediate queue that is not assigned to an smsc until the smsc queue is less than 5min on the spscific smsc. This way in case of a restart, the 1st smsc to be available will get a moderate queue and the rest of smsc's will loadbalace the rest of messages once. Would something like that might be possible? I fear that an approach like this one will require a big remake of kannel queue/load management and I doubt kannel team will go for it. I can do that. No problem. But you don't need a patch for that, you can use configuration. Have you tried setting in smsc group: sms-outgoing-queue-limit = 50? This will push the first 50 sms to your smsc, and then push single sms, until your other smscs go online. Also I stick in the opinion that kannel shouldn't panic if it can't set active/enable an smsc on startup since there can be other smscs that can process messages for other routes/destinations. This issue is more important since the only workarroud is to edit kannel.conf comment the offending smsc and restart kannel and once the smsc is fixed, re-edit kannel.conf and restart kannel since so far, there is no way to make kannel reread the config file. We are in total agreement. I never liked this issue myself, either (both failing to startup and not reconfiguring on the fly). Once I test the patch I'll let you know. Probably this weekend. Please don't. Wait for the patch update. Regards and THANKS for your time Support. No pro, bro. Sorry for the delay, and for your ordeal. Mexico should be beautiful this time of the year. Alvaro |-| EnvΓe y Reciba Datos y mensajes de Texto (SMS) hacia y desde cualquier celular y Nextel en el PerΓΊ, MΓ©xico y en mas de 180 paises. Use aplicaciones 2 vias via SMS y GPRS online Visitenos en www.perusms.NET www.smsglobal.com.mx y www.pravcom.com On Tue, Jun 9, 2009 at 10:14 PM, Alan McNattya...@catalyst.net.nz wrote: Hi Nikos, OK, yeah I'm not about failure at start-up time. If an SMSC is off-line (SMPP timeout/retry during bind) kannel will start (simply config a bogus IP address for SMPP to test this out). An example is lets say we have 2 connections for load balancing. Now consider 1 smsc being taken offline (no tcp connection) as it is upgraded, we can still send through the other (this is also useful for DR - disaster recover, etc). So from my example above only 1 connection would be active the other in bind retry loop (wait X seconds, etc) - no tcp connection is established to smsc. If we re-started kannel it wouldn't be a good idea to wait for both links to become active before pushing messages out - this would mean waiting for the upgraded smsc to
Re: [PATCH] Guaranteed throughput smsc-independent
On Wed, Jun 10, 2009 at 09:51:08AM +0300, Nikos Balkanas wrote: Hi Donald, I think you hit the nail exactly on the head. AFAIK, there is a single thread at this point, sms_router, that is implementing bb_smscconn_sent. Therefore all sms in the outgoing smscs should be delayed. Kannel doesn't raise threads on demand. Sorry I don't get your point, each smsc is delay on it's own according to my tests (4 fake smsc with 10sms/s load balanced). There may be an additional issue. This approach, which delays everything by a set amount if there is a throughput constraint, will utilize exactly half the Uh? available bandwidth. If for example an smsc has a throughput of 10 sms/1', and there is a queue of 20 sms, In the old way, assuming it works correctly, the first 10 should be send at once at time 0, and the rest at atime 1' with an average delay of 30/sms. With this approach, 1 sms would be sent every 6, with an average delay of 1'/sms. Sorry, I don't get this, could you maybe explain it in another way? If you got 20 sms in the queue and a throughput of 10, they'll be send in 2 seconds. 10 during the first second and 2 during the second second ;). Even more, with that algorithm you might take a sample of length N seconds and you should only find throughput*N outgoing messages (with each message separated by 1/throughput seconds). Coupled with the previous point, it could spell problems for queues in large installations that use many smscs to load balance. The advantage is that it offers solid throughput handling. I am a wap person, so I leave this decision for peoaple with a lot of smss. I'll be answering the other mails soon. Thanks for your input. Damián Viano(Des).
Re: Kannel not load balancing after restart and messages get stuckinqueue
then it's a bug ion smsc_at module and must be fixed.
Please report it via redmine and I will look into it when time
permits...
Thanks,
Alex
Am 10.06.2009 um 16:02 schrieb Alvaro Cornejo:
Alex
If so, why if I have a runing kannel with all smsc's up (modems) and
one of them died/get locked and restart kannel it will panic and the
config haven't changed ?
If do nothing I get a reconnecting status on kannel and all other
smsc's continue to work normally. I can even disconnect the modem and
replace it and kannel will reenable the smsc. Kannel will continue to
work properly. Also I can shutdown/up the smsc wih no problem.
The panic occurs if I do reset kannel with a locked or disconnected
modem.
Regards
Alvaro
|
-|
Envíe y Reciba Datos y mensajes de Texto (SMS) hacia y desde cualquier
celular y Nextel
en el Perú, México y en mas de 180 paises. Use aplicaciones 2 vias via
SMS y GPRS online
Visitenos en www.perusms.NET www.smsglobal.com.mx y
www.pravcom.com
On Wed, Jun 10, 2009 at 8:45 AM, Alexander
Malyshamal...@kannel.org wrote:
Hi Nikos,
this panic is reached only if SMSC module has wrong configuration.
This panic has nothing todo with active/offline/not connected SMSCs.
Thanks,
Alex
Am 10.06.2009 um 14:32 schrieb Nikos Balkanas:
Hi Milan,
Please edit gw/bb_smsccconn.c and go to smsc2_start:
for (i = 0; i gwlist_len(smsc_groups) .
(grp = gwlist_get(smsc_groups, i)) != NULL; i++) {
conn = smscconn_create(grp, 1);.
if (conn == NULL)
panic(0, Cannot start with SMSC connection failing);
gwlist_append(smsc_list, conn);
}
That's panic. That's what I am talking about.
I didn't write these rules, and l am sorry, I don't have the time to
continue this debate.
BR,
Nikos
- Original Message - From: Milan P. Stanic m...@arvanta.net
To: devel@kannel.org
Sent: Wednesday, June 10, 2009 2:55 PM
Subject: Re: Kannel not load balancing after restart and messages
get
stuckinqueue
On Wed, 2009-06-10 at 14:32, Nikos Balkanas wrote:
I happen to agree also. But the fact is, that this is how kannel
currently works. And i have to abide by that. Anyway, read my
response
to
Alvaro, it may be just a configuration issue.
I'm not sure if we are talking about same behavior of bearerbox
but as
I see it, bearerbox in 1.4.3 version (and some previous) starts and
stays even if no one smsc (smsbox can't connect to SMSC) is
running or in
online status.
If you talk about some specific configuration (smsc routing or load
balancing) then you might be right. I didn't tested such use cases.
Or, I misunderstood your post totally.
- Original Message - From: Alan McNatty a...@catalyst.net.nz
To: nbalka...@gmail.com
Cc: devel@kannel.org
Sent: Wednesday, June 10, 2009 2:23 PM
Subject: Re: Kannel not load balancing after restart and
messages get
stuck inqueue
Milan P. Stanic wrote:
On Wed, 2009-06-10 at 10:02, Nikos Balkanas wrote:
I must be the one thanking you for the discussion. You see I
am a wap
guy, and don't know of many of the issues that sms people face.
The only concern is at startup. After it has bbox has started
succesfully, and therefore sms_router, started as well, it
doesn't
matter. You can take all the smscs you want offline to
service them.
sms_router will work with the rest of them.
The only issue is for startup. I cannot test the scenario you
describe,
because i don't have real smsc connections. However, if both
connections
are active the minute kannel starts (1st case0, there should
be no
problem. In the second case, the unavailable smsc should be
commented out
the configuration.
I guess the only question is: Under what circumstances bbox
will
start,
if the 2nd smsc is not responding?
Bearerbox should always start. Even if there is no one smsc
active.
Smsc can become active later for miscellaneous reasons.
I agree 100% with Milan here. SMSC's are not in our control -
there
could be many points of failure between kannel and SMSC.
Firewalls,
VPN's etc may be unhappy and we lose connectivity .. we need
kannel to
be stable (in operation and restart, etc) and manage the
reconnect and
message distribution (SMPP is pretty good here).
As Alvaro was suggesting earlier in the thread some sort of
dynamic
redistribution of queued message (maybe like keepalived or
sim.) could
be ideal .. but obviously some work ..
Nikos I appreciate you have been busy doing some development. I
can
help
test some of the SMSC 'stuff' (from SMPP) perspective - it's
obviously
in my interest here ;). As of now I haven't used a-lot of the
routing
features so this is a bit new to me (I will try and get back
into it
and
hopefully make some suggestions).
--
Kind regards, Milan
Re: [PATCH] Guaranteed throughput smsc-independent
On Wed, Jun 10, 2009 at 09:38:41AM +0200, Alexander Malysh wrote: Hi, not only this is issue. You want not sleep! Just think about it a bit. Sleeping was the current approach so I just followed it. I think the idea behind that (and behind the code to ensure the sleeping time is accomplished) is that the smsc have different threads for TX(MTs) and RX(MOs) so making the TX thread sleep shouldn't be a problem, this might be a false assumption. Hint: you want to process MOs and ACKs without delays... -1 Could you point me to some code affected by this? And please don't just tell me that the way throughput is currently implemented is wrong and therefore the patch I made to fix the current implementation is also wrong. I'd really appreciate it, since I'm only trying to help by fixing a problem. If the current (sleeping) approach is wrong, great, let's talk about it and find another way to implement it better. Thanks, Alex P.S. I already posted patch that fixes this issue but not in ideal way. SMSC will not sleep but sending all allowed messages in the time slot at full speed instead of delaying each message. Full speed? I'm not sure I get that point keeping in mind that if you send for a 10 sms/s throughput, 20 sms in 1 second and then nothing for another second, you still violated the throughput, as you sent 20sms/s. Thanks for your input on my patch, I appreciate it, and hope you have time to further explain your points. Damián Viano(Des).
Re: [PATCH] Guaranteed throughput smsc-independent
On Wed, Jun 10, 2009 at 11:20:11AM +0300, Nikos Balkanas wrote: Actually, I don't think that even this patch can guarantee solid throughput. All I/O (including SMSc) is handled by a polling thread. AFAIK bb_smscconn_sent just puts the sms in the connection queue, where the What? No. bb_smscconn_sent just account for a sent sms and generate the corresponding DLR, AND is an MUST callback for the smsc after a successful send of a message which is the reason that motivated me to assume that the smsc sender thread will call this function (confirmed by the fakesmsc). connection layer takes over and sends it. If there is a queue for that SMSc, the connection layer will still try to empty it as fast as it can. IMHO, any robust throughput should be implemented in the connection layer. The idea behind this patch is to avoid implementing it for every connection, without loosing the fact that the affected layer is the connection layer. That's exactly the reason I choose this callback to do the sleeping and no other part. BR, Nikos I hope I clarified your doubts about my patch, and I welcome any further input. Thanks for your point of view, I think I failed to inform better why this is supposed to work and your mail make me write it a lot clearer (I hope!). Damián Viano(Des).
Re: [PATCH] Guaranteed throughput smsc-independent
Am 10.06.2009 um 16:40 schrieb Damian Viano: On Wed, Jun 10, 2009 at 09:38:41AM +0200, Alexander Malysh wrote: Hi, not only this is issue. You want not sleep! Just think about it a bit. Sleeping was the current approach so I just followed it. I think the idea behind that (and behind the code to ensure the sleeping time is accomplished) is that the smsc have different threads for TX(MTs) and RX(MOs) so making the TX thread sleep shouldn't be a problem, this might be a false assumption. false assumption, TX and MX may running in one thread (e.g. SMPP v3.4 transceiver mode). Hint: you want to process MOs and ACKs without delays... -1 Could you point me to some code affected by this? And please don't just tell me that the way throughput is currently implemented is wrong and therefore the patch I made to fix the current implementation is also wrong. see above.. and yes, current implementation is wrong. and why should we accept fix for a wrong version instead of doing it right ? I'd really appreciate it, since I'm only trying to help by fixing a problem. If the current (sleeping) approach is wrong, great, let's talk about it and find another way to implement it better. something like this: http://github.com/amalysh/kannel/commit/31db912a9e5aecd6d9f038b56d09f408f2c256f9 Thanks, Alex P.S. I already posted patch that fixes this issue but not in ideal way. SMSC will not sleep but sending all allowed messages in the time slot at full speed instead of delaying each message. Full speed? I'm not sure I get that point keeping in mind that if you send for a 10 sms/s throughput, 20 sms in 1 second and then nothing for another second, you still violated the throughput, as you sent 20sms/s. Thanks for your input on my patch, I appreciate it, and hope you have time to further explain your points. Damián Viano(Des).
Re: [PATCH] Guaranteed throughput smsc-independent
On Wed, Jun 10, 2009 at 03:06:59PM +0200, Alexander Malysh wrote: here is a link to throttling branch: http://github.com/amalysh/kannel/tree/smpp-throttling only smpp implemented for now. Great, thanks for the link. I see that you basically break out when you are over throughput[1], I see your concern now, since the thread for receiving is the same as the sender, so you just break out to keep processing incoming pdu. Makes sense, and this indeed breaks my assumption that the smsc implementations would have a thread for sending and another one for receiving. Unfortunately this is smsc-dependant, so maybe we could move this to a more generic api, and make the rule that the SMSCs MUST check that api (say... conn-over_throughput()?) and move the load setup/increase to the already existing callbacks?. That way it's a bit more generic and we can implement conn-sleep_for_throughput() for the smsc that need their sender thread to sleep, or let the others (i.e. smpp) handle it accordingly. Opinions about this approach? [1] I didn't check the load object but I assume it account for load_increases in the set interval, which would make sense. Thanks, Alex Thanks your code, made your points a lot more clearer, I hope we can come up with a better implementation of throughput for all kannel users. Damián Viano(Des).
Re: [PATCH] Guaranteed throughput smsc-independent
On Wed, Jun 10, 2009 at 05:00:21PM +0200, Alexander Malysh wrote: false assumption, TX and MX may running in one thread (e.g. SMPP v3.4 transceiver mode). see above.. and yes, current implementation is wrong. and why should we accept fix for a wrong version instead of doing it right ? something like this: http://github.com/amalysh/kannel/commit/31db912a9e5aecd6d9f038b56d09f408f2c256f9 I've answered all this on the mail just sent in reply to your link to your branch on github, I already checked your code, and came to this same conclusions. Damián Viano(Des).
Re: LMT extensions
You can checkout the cvs version of kannel which is able to handle customized TLVs on smpp... Julien 2009/6/9 V ve...@neonet.lt: Hello, Yes, we already run kannel with meta-data/tlv support (cvs branch). But, it's strange, why LMT use some PDU, that are non standart. On what conditions we could get that patch? regards, Martynas Hi, LMT uses SMPP meta-data optional parameters, its easy to implement it using meta-data build. LMT additionally uses two specific PDU's (LMT_REPORT and LMT_REPORT_RESP) what must be programmed directly into Kannel (those are actually LMT specific DLR PDU's and can be mapped to DLR requests in Kannel) We are using cvs-meta-data build with LMT patch, works correctly. Illimar Hi, without their Spec is this impossible to say if Kannel can handle this... Please provide Spec. Thanks, Alex Am 09.06.2009 um 10:11 schrieb V: Hello, Any have some experience with connection to LMT operator in Latvia with proprietary SMPP extensions for DLR and billing reports? I found previous thread about it, but with no decisions..:( Kind regards, Martynas - Hi, Am I right that LMT uses reserved command ids there? If yes, then in my opinion it may be useful to implement some API for adding vendor specific SMPP commands processing as DSO and after this LMT extension as such DSO :-) Illimar Reinbusch - Telejazz.com wrote: LMT uses SMPP procol with some additions to it for billing and DLR purpose. * *For example:* * **LMT_REPORT – Contains information regarding the result of the processing and execution of the SUBMIT_SM command;* *LMT_GEN_CODE_RESP – Contains information regarding the result of the processing and execution of the LMT_GEN_CODE command** LMT_VERIFY_CODE_RESP – Contains information regarding the result of the processing and execution of the LMT_ VERIFY_CODE command; * *LMT_REPORT_RESP – contains information regarding the results of the LMT_REPORT command’s execution;* *LMT_GEN_CODE – requests to generate access information for the Service provider’s public resource, if authorization is made according to the procedure outlined in Art.3.7. and 3.8. of the Addendum No.2 to this Contract;* *LMT_VERIFY_CODE – requests to verify the access information provided by the Client for the public resource; Illimar Hi, what is LMT extention? Could you please provide some links? Illimar Reinbusch - Telejazz.com wrote: Have somebody implemented it or its new to Kannel. Would Kannel be interested adding support for it to Kannel if someone (me) would write it? Illimar Jurijs Cerepanovs wrote: Illimar Reinbusch - Telejazz.com wrote: Current implementation needs patch for kannel - it's simply to rewrite smpp connector for understanding LMT_REPORT e.t.c proprietary packets. Hi Has anybody implemented or added new protocol to support Latvia LMT operator SMPP protocol with LMT extentsions? Would u suggest adding to to current SMPP development or add new protocol based on SMPP implementation? Thanks, Illimar -- Michael Bochkaryov www.netstyle.com.ua -- Julien Buratto
Re: [PATCH] Guaranteed throughput smsc-independent
Am 10.06.2009 um 17:16 schrieb Damian Viano: On Wed, Jun 10, 2009 at 03:06:59PM +0200, Alexander Malysh wrote: here is a link to throttling branch: http://github.com/amalysh/kannel/tree/smpp-throttling only smpp implemented for now. Great, thanks for the link. I see that you basically break out when you are over throughput[1], I see your concern now, since the thread for receiving is the same as the sender, so you just break out to keep processing incoming pdu. Makes sense, and this indeed breaks my assumption that the smsc implementations would have a thread for sending and another one for receiving. Unfortunately this is smsc-dependant, so maybe we could move this to a more generic api, and make the rule that the SMSCs MUST check that api (say... conn-over_throughput()?) and move the load setup/increase to the already existing callbacks?. I think, it's already enough abstracted in load object. Now it's SMSC dependent how to use it. So I don't think we need more api there but I'm open for a patch that would make it easier to implement this in other smsc modules. That way it's a bit more generic and we can implement conn-sleep_for_throughput() for the smsc that need their sender thread to sleep, or let the others (i.e. smpp) handle it accordingly. Opinions about this approach? [1] I didn't check the load object but I assume it account for load_increases in the set interval, which would make sense. Thanks, Alex Thanks your code, made your points a lot more clearer, I hope we can come up with a better implementation of throughput for all kannel users. Damián Viano(Des).
Re: [PATCH] Guaranteed throughput smsc-independent
Hi, I actually didn't realise that bb_smsccon_sent is a callback, and kept mixing it up with smscconn_sent. Plus you state that each SMSc connection has its own thread, and Donald agrees. That's much better. You shouldn't have to explain it, I should have known better. I am a wap person and haven't been involved with SMS that much. I would need to study the architecture better at this point, before sticking my foot in my mouth again. BR, Nikos - Original Message - From: Damian Viano damian.vi...@buongiorno.com To: Nikos Balkanas nbalka...@gmail.com Cc: Donald Jackson donaldjs...@gmail.com; devel@kannel.org Sent: Wednesday, June 10, 2009 5:50 PM Subject: Re: [PATCH] Guaranteed throughput smsc-independent On Wed, Jun 10, 2009 at 11:20:11AM +0300, Nikos Balkanas wrote: Actually, I don't think that even this patch can guarantee solid throughput. All I/O (including SMSc) is handled by a polling thread. AFAIK bb_smscconn_sent just puts the sms in the connection queue, where the What? No. bb_smscconn_sent just account for a sent sms and generate the corresponding DLR, AND is an MUST callback for the smsc after a successful send of a message which is the reason that motivated me to assume that the smsc sender thread will call this function (confirmed by the fakesmsc). connection layer takes over and sends it. If there is a queue for that SMSc, the connection layer will still try to empty it as fast as it can. IMHO, any robust throughput should be implemented in the connection layer. The idea behind this patch is to avoid implementing it for every connection, without loosing the fact that the affected layer is the connection layer. That's exactly the reason I choose this callback to do the sleeping and no other part. BR, Nikos I hope I clarified your doubts about my patch, and I welcome any further input. Thanks for your point of view, I think I failed to inform better why this is supposed to work and your mail make me write it a lot clearer (I hope!). Damiαn Viano(Des).
Re: [PATCH] Guaranteed throughput smsc-independent
On Wed, Jun 10, 2009 at 05:28:04PM +0200, Alexander Malysh wrote:
Am 10.06.2009 um 17:16 schrieb Damian Viano:
On Wed, Jun 10, 2009 at 03:06:59PM +0200, Alexander Malysh wrote:
here is a link to throttling branch:
http://github.com/amalysh/kannel/tree/smpp-throttling
only smpp implemented for now.
Great, thanks for the link.
I see that you basically break out when you are over throughput[1], I
see your concern now, since the thread for receiving is the same as
the sender, so you just break out to keep processing incoming pdu.
Makes sense, and this indeed breaks my assumption that the smsc
implementations would have a thread for sending and another one for
receiving.
However there's still a problem. In the case of smpp not in transceiver
mode the io_thread() code that calls send_messages() is basically:
for (;conn != NULL;) {
while ((ret = read_pdu(smpp, conn, len, pdu)) == 1) { //nothing since we
don't read
}
...
if (transmitter difftime(time(NULL), smpp-throttling_err_time)
SMPP_THROTTLING_SLEEP_TIME) {
smpp-throttling_err_time = 0;
send_messages(smpp, conn, pending_submits);
}
}
So if send_messages() does not sleep and we are over the throughput this
basically comes down to a tight loop that in transceiver is controlled
by read_pdu() blocking on the network read.
Have you considered that?
I'm starting to think that we'll need to know 1) if we are over the throughput
2) if we are in some kind of transceiver mode or not.
Unfortunately this is smsc-dependant, so maybe we could move this to
a more generic api, and make the rule that the SMSCs MUST check that
api (say... conn-over_throughput()?) and move the load
setup/increase to the already existing callbacks?.
I think, it's already enough abstracted in load object. Now it's SMSC
dependent how to use it.
So I don't think we need more api there but I'm open for a patch that
would make it easier to implement
this in other smsc modules.
That way it's a bit more generic and we can implement
conn-sleep_for_throughput() for the smsc that need their sender
thread to sleep, or let the others (i.e. smpp) handle it accordingly.
Opinions about this approach?
[1] I didn't check the load object but I assume it account for
load_increases in the set interval, which would make sense.
Thanks,
Alex
Thanks your code, made your points a lot more clearer, I hope we can
come up
with a better implementation of throughput for all kannel users.
Damián Viano(Des).
Re: [PATCH] Guaranteed throughput smsc-independent
This patch was only to show how it could be done. It's not complete
for SMPP.
For SMPP the main loop should be reordered. I have local version that
works
as expected but I need some time to extract it because of many
differences.
I will try to extract it this weekend...
Thanks,
Alex
Am 10.06.2009 um 21:51 schrieb Damian Viano:
On Wed, Jun 10, 2009 at 05:28:04PM +0200, Alexander Malysh wrote:
Am 10.06.2009 um 17:16 schrieb Damian Viano:
On Wed, Jun 10, 2009 at 03:06:59PM +0200, Alexander Malysh wrote:
here is a link to throttling branch:
http://github.com/amalysh/kannel/tree/smpp-throttling
only smpp implemented for now.
Great, thanks for the link.
I see that you basically break out when you are over
throughput[1], I
see your concern now, since the thread for receiving is the same as
the sender, so you just break out to keep processing incoming pdu.
Makes sense, and this indeed breaks my assumption that the smsc
implementations would have a thread for sending and another one for
receiving.
However there's still a problem. In the case of smpp not in
transceiver
mode the io_thread() code that calls send_messages() is basically:
for (;conn != NULL;) {
while ((ret = read_pdu(smpp, conn, len, pdu)) == 1) { //nothing
since we don't read
}
...
if (transmitter difftime(time(NULL), smpp-throttling_err_time)
SMPP_THROTTLING_SLEEP_TIME) {
smpp-throttling_err_time = 0;
send_messages(smpp, conn, pending_submits);
}
}
So if send_messages() does not sleep and we are over the throughput
this
basically comes down to a tight loop that in transceiver is controlled
by read_pdu() blocking on the network read.
Have you considered that?
I'm starting to think that we'll need to know 1) if we are over the
throughput
2) if we are in some kind of transceiver mode or not.
Unfortunately this is smsc-dependant, so maybe we could move this to
a more generic api, and make the rule that the SMSCs MUST check that
api (say... conn-over_throughput()?) and move the load
setup/increase to the already existing callbacks?.
I think, it's already enough abstracted in load object. Now it's SMSC
dependent how to use it.
So I don't think we need more api there but I'm open for a patch that
would make it easier to implement
this in other smsc modules.
That way it's a bit more generic and we can implement
conn-sleep_for_throughput() for the smsc that need their sender
thread to sleep, or let the others (i.e. smpp) handle it
accordingly.
Opinions about this approach?
[1] I didn't check the load object but I assume it account for
load_increases in the set interval, which would make sense.
Thanks,
Alex
Thanks your code, made your points a lot more clearer, I hope we can
come up
with a better implementation of throughput for all kannel users.
Damián Viano(Des).
