RE: repost: [PATCH] SMPP: data_coding (deliver_sm) votes?!
On Fri, 2002-11-22 at 04:54, Alan McNatty wrote: Hi Oded, It was included - but as mentioned implied iconv was required not optional. No so - configure will check for it and set HAVE_ICONV only if the header file exists (which to me suggests that the library is installed - I don't know many people who hold on their build systems headers for libraries they don't have installed. It sounds like the preference would be to have as optional (propagate the code with a few ifdef's, etc) and default to gsm_to_latin1 conversion regardless of encoding (will look at this shortly). Don't think that's a correct approach. coding anything to latin1 regardless of the SMSC's preferences or the user's preferences just because they user did not specificly enable a feature which should be auto-detected does not sound like a good idea to me. -- Oded Arbel m-Wise mobile solutions __ Scanned and protected by Inflex and McAfee AntiVirus
Re: repost: [PATCH] SMPP: data_coding (deliver_sm) votes?!
On Sun, 2002-11-24 at 05:34, Stipe Tolj wrote: I would leave it in gsm character set as far as possible because the receiver might want to do its own conversion. And converting it to latin1 is not lossless. in any case not convert it if coding means its binary yep, agreed, binary things should never be converted. Ahmm.. may I point to the fact that the patch Alan submitted already does these two last discussed issues: - convert using iconv when possible - warn when failing - allows setting binary to prevent encoding of default data coding The only thing it does not do is to warn if encoding is needed and iconv is not available. this could be easily fixed by adding an #else section to the charset_convert body's #ifdef, that try to do conversion using libxml and outputs an error if it fails. if no one will beat me to it, I can have a patch that does it in a couple of days. -- Oded Arbel m-Wise mobile solutions __ Scanned and protected by Inflex and McAfee AntiVirus
Re: repost: [PATCH] SMPP: data_coding (deliver_sm) votes?!
On Sonntag, November 24, 2002, at 04:35 Uhr, Alan McNatty wrote: On Fri, 2002-11-22 at 21:04, Andreas Fink wrote: Why gsm_to_latin1 if we know encoding? We don't, well the smpp driver doesn't - that's the whole point of this patch. In the case where the data coding is set to SMSC default (0x00) - the smpp driver doesn't know what decoding to do. Well the spec says that its GSM character set. We need to be able to specify via the config what encoding each smsc coonsiders to be it's default - they do vary. Ok. that would be a config variable telling which one to be considered. Currently all incomming are assumed to gsm hence gsm_to_latin1 encoding is performed on all incomming messages from all smsc via smpp driver. However, the SMSC default encoding could easily be ASCII, for example. yes but what I'm saying that is we should leave it in the character set it is. gsm_to_latin1 is something which is not a good idea either as it is run also on binary and unicode content in the current code. In a situation where you chain multiple kannels for incoming you simply want to forward the incoming SMS "as is" without modifiying it whatsoever. indicating what encodig default means is ok but we should not in any way modifiy the data at this point. We should maybe do it in smsbox where we forward it to a webpage and indicate there which character set the webpage accepts and then convert it once. We want to avoid situations where we do gsm character set -> latin1 -> gsm character set as some characters would not pass this way. -- consider -- If iconv is not available or no default is set via config and the data coding is smsc default (0x00) - we assume gsm encoding so the default behaviour is the same as current. Alternatively, if iconv is not available and data coding requires iconv or data coding is set via config to something which would require iconv, we sould error and do no conversion (because we can't). Otherwise - we utilise Oded's work with iconv to do sensible decoding of incomming message. the idea is good but the SMPP driver is the wrong place do to this. the only thing we should do there is to say what is considered the default so we can convert it if needed when the message leaves kannel. Andreas Fink Global Networks, Inc. -- Tel: +41-61-333 Fax: +41-61-6932729 Mobile: +41-79-2457333 Global Networks, Inc. Clarastrasse 3, 4058 Basel, Switzerland Web: http://www.global-networks.ch/ [EMAIL PROTECTED] -- Member of the GSM Association
Re: repost: [PATCH] SMPP: data_coding (deliver_sm) votes?!
Hi At 12:57 PM 11/24/02 +0100, Andreas Fink wrote: We don't, well the smpp driver doesn't - that's the whole point of this patch. In the case where the data coding is set to SMSC default (0x00) - the smpp driver doesn't know what decoding to do. Well the spec says that its GSM character set. actually not, quote from smpp 3.4 specs 5.2.19 data_coding Bits 7 6 5 4 3 2 1 0 Meaning Notes 0 0 0 0 0 0 0 0 SMSC Default Alphabet But no mention of what the default alphabet of the smsc ought tio be. nisan
RE: repost: [PATCH] SMPP: data_coding (deliver_sm) votes?!
yes but what I'm saying that is we should leave it in the character set it is. gsm_to_latin1 is something which is not a good idea either as it is run also on binary and unicode content in the current code. In a situation where you chain multiple kannels for incoming you simply want to forward the incoming SMS "as is" without modifiying it whatsoever. indicating what encodig default means is ok but we should not in any way modifiy the data at this point. We should maybe do it in smsbox where we forward it to a webpage and indicate there which character set the webpage accepts and then convert it once. We want to avoid situations where we do gsm character set - latin1 - gsm character set as some characters would not pass this way. That's sounds about right.if you also encode from external characater set to GSM or UCS2 in smsbox on the way in (MT), then I'm all for moving all the charset conversions to smsbox. one point though - how will you handle drivers (like SMPP) that allow you to send specific native character encodings instead of UCS-2 when using non-GSM compatible alphabets ? send only in unicode ? --Oded Arbelm-Wise mobile solutions[EMAIL PROTECTED] +972-9-9581711 (116)+972-67-340014 ::..May's Law:The quality of correlation is inversly proportional to the densityof control. (The fewer the data points, the smoother the curves.) __ Scanned and protected by Inflex and McAfee AntiVirus
Re: repost: [PATCH] SMPP: data_coding (deliver_sm) votes?!
On Sonntag, November 24, 2002, at 03:36 Uhr, Oded Arbel wrote: yes but what I'm saying that is we should leave it in the character set it is. gsm_to_latin1 is something which is not a good idea either as it is run also on binary and unicode content in the current code. In a situation where you chain multiple kannels for incoming you simply want to forward the incoming SMS "as is" without modifiying it whatsoever. indicating what encodig default means is ok but we should not in any way modifiy the data at this point. We should maybe do it in smsbox where we forward it to a webpage and indicate there which character set the webpage accepts and then convert it once. We want to avoid situations where we do gsm character set -> latin1 -> gsm character set as some characters would not pass this way. That's sounds about right. if you also encode from external characater set to GSM or UCS2 in smsbox on the way in (MT), then I'm all for moving all the charset conversions to smsbox. one point though - how will you handle drivers (like SMPP) that allow you to send specific native character encodings instead of UCS-2 when using non-GSM compatible alphabets ? send only in unicode ? In that case the coding should indicate the specific native character set and the recoding should happen in SMSbox if indicated accordingly. Andreas Fink Global Networks, Inc. -- Tel: +41-61-333 Fax: +41-61-6932729 Mobile: +41-79-2457333 Global Networks, Inc. Clarastrasse 3, 4058 Basel, Switzerland Web: http://www.global-networks.ch/ [EMAIL PROTECTED] -- Member of the GSM Association
RE: repost: [PATCH] SMPP: data_coding (deliver_sm) votes?!
one point though - how will you handle drivers (like SMPP) that allow you to send specific native character encodings instead of UCS-2 when using non-GSM compatible alphabets ? send only in unicode ? In that case the coding should indicate the specific native character set and the recoding should happen in SMSbox if indicated accordingly. I was talking about MT - for example, you get a hebrew message, so SMSBox encodes it to UCS2 and sends it to Bearerbox which decides the proper connection is through anSMPP driver (for example). the SMPP driver for version 3.4 supports sending 8 bit hebrew characters in the SM by setting the proper data_coding flag. can we handle that ? --Oded Arbelm-Wise mobile solutions[EMAIL PROTECTED] +972-9-9581711 (116)+972-67-340014 ::..Self-confidence isn't everything, its the only thing. __ Scanned and protected by Inflex and McAfee AntiVirus
Re: repost: [PATCH] SMPP: data_coding (deliver_sm) votes?!
On Sonntag, November 24, 2002, at 04:49 Uhr, Oded Arbel wrote: one point though - how will you handle drivers (like SMPP) that allow you to send specific native character encodings instead of UCS-2 when using non-GSM compatible alphabets ? send only in unicode ? In that case the coding should indicate the specific native character set and the recoding should happen in SMSbox if indicated accordingly. I was talking about MT - for example, you get a hebrew message, so SMSBox encodes it to UCS2 and sends it to Bearerbox which decides the proper connection is through an SMPP driver (for example). the SMPP driver for version 3.4 supports sending 8 bit hebrew characters in the SM by setting the proper data_coding flag. can we handle that ? well for MT messages it has to be done in the driver or in bearerbox. Andreas Fink Global Networks, Inc. -- Tel: +41-61-333 Fax: +41-61-6932729 Mobile: +41-79-2457333 Global Networks, Inc. Clarastrasse 3, 4058 Basel, Switzerland Web: http://www.global-networks.ch/ [EMAIL PROTECTED] -- Member of the GSM Association
RE: repost: [PATCH] SMPP: data_coding (deliver_sm) votes?!
one point though - how will you handle drivers (like SMPP) that allow you to send specific native character encodings instead of UCS-2 when using non-GSM compatible alphabets ? send only in unicode ? In that case the coding should indicate the specific native character set and the recoding should happen in SMSbox if indicated accordingly. I was talking about MT - for example, you get a hebrew message, so SMSBox encodes it to UCS2 and sends it to Bearerbox which decides the proper connection is through anSMPP driver (for example). the SMPP driver for version 3.4 supports sending 8 bit hebrew characters in the SM by setting the proper data_coding flag. can we handle that ? well for MT messages it has to be done in the driver or in bearerbox. That's fine, but if the SMSBox does all the character set encoding, then the driver has no clue as to what was the original encoding. of course it can simply try to encode in all the supported charsets to see what succeeds, but this is hardly an optimal solution. --Oded Arbelm-Wise mobile solutions[EMAIL PROTECTED] +972-9-9581711 (116)+972-67-340014 ::.."Only wimps use tape backups; real men put their software on ftp-serversand let the rest of the world mirror it." -- Linus Torvalds __ Scanned and protected by Inflex and McAfee AntiVirus
Re: repost: [PATCH] SMPP: data_coding (deliver_sm) votes?!
On Fri, 2002-11-22 at 21:04, Andreas Fink wrote: Why gsm_to_latin1 if we know encoding? We don't, well the smpp driver doesn't - that's the whole point of this patch. In the case where the data coding is set to SMSC default (0x00) - the smpp driver doesn't know what decoding to do. We need to be able to specify via the config what encoding each smsc coonsiders to be it's default - they do vary. Currently all incomming are assumed to gsm hence gsm_to_latin1 encoding is performed on all incomming messages from all smsc via smpp driver. However, the SMSC default encoding could easily be ASCII, for example. -- consider -- If iconv is not available or no default is set via config and the data coding is smsc default (0x00) - we assume gsm encoding so the default behaviour is the same as current. Alternatively, if iconv is not available and data coding requires iconv or data coding is set via config to something which would require iconv, we sould error and do no conversion (because we can't). Otherwise - we utilise Oded's work with iconv to do sensible decoding of incomming message. Sound reasonable? Cheers, Alan
Re: repost: [PATCH] SMPP: data_coding (deliver_sm) votes?!
I would leave it in gsm character set as far as possible because the receiver might want to do its own conversion. And converting it to latin1 is not lossless. in any case not convert it if coding means its binary yep, agreed, binary things should never be converted. Stipe [EMAIL PROTECTED] --- Wapme Systems AG Vogelsanger Weg 80 40470 Düsseldorf Tel: +49-211-74845-0 Fax: +49-211-74845-299 E-Mail: [EMAIL PROTECTED] Internet: http://www.wapme-systems.de --- wapme.net - wherever you are
Re: repost: [PATCH] SMPP: data_coding (deliver_sm) votes?!
On Freitag, November 22, 2002, at 03:54 Uhr, Alan McNatty wrote: Hi Oded, It was included - but as mentioned implied iconv was required not optional. It sounds like the preference would be to have as optional (propagate the code with a few ifdef's, etc) and default to gsm_to_latin1 conversion regardless of encoding (will look at this shortly). Why gsm_to_latin1 if we know encoding? I would leave it in gsm character set as far as possible because the receiver might want to do its own conversion. And converting it to latin1 is not lossless. in any case not convert it if coding means its binary Andreas Fink Global Networks, Inc. -- Tel: +41-61-333 Fax: +41-61-6932729 Mobile: +41-79-2457333 Global Networks, Inc. Clarastrasse 3, 4058 Basel, Switzerland Web: http://www.global-networks.ch/ [EMAIL PROTECTED] -- Member of the GSM Association
RE: repost: [PATCH] SMPP: data_coding (deliver_sm) votes?!
Hi Oded, It was included - but as mentioned implied iconv was required not optional. It sounds like the preference would be to have as optional (propagate the code with a few ifdef's, etc) and default to gsm_to_latin1 conversion regardless of encoding (will look at this shortly). Given that even libxml requires iconv for full support - it would be nice to at least 'recommend' iconv when compiling kannel esp considering things may not work as they should in smpp world for some. On Fri, 2002-11-22 at 03:23, Oded Arbel wrote: The original charset_convert patch had a patch for configure. it should have been in the current patch - I'll look into it again. -- Oded Arbel m-Wise mobile solutions [EMAIL PROTECTED] +972-9-9581711 (116) +972-67-340014 ::.. There is no spoon. (only annoying little kids) ! -Original Message- From: Stipe Tolj [mailto:[EMAIL PROTECTED]] Sent: Thursday, November 21, 2002 2:10 AM To: Oded Arbel Cc: Andreas Fink; [EMAIL PROTECTED] Subject: Re: repost: [PATCH] SMPP: data_coding (deliver_sm) votes?! the patch tests on the header files, not on the library itself. BTW, the patch does not work when linking the executables for me, because configure has not added any lib dependecy statements for ld. Ok, I'd propose the following behaviour: configure tests on iconv presence. If available sets #define HAVE_ICONV in config.h and the iconv routines are used directly in the code. If not available use the old behaviour. What about that approach? Stipe [EMAIL PROTECTED] --- Wapme Systems AG Vogelsanger Weg 80 40470 Düsseldorf Tel: +49-211-74845-0 Fax: +49-211-74845-299 E-Mail: [EMAIL PROTECTED] Internet: http://www.wapme-systems.de --- wapme.net - wherever you are -- Alan McNatty -- Catalyst IT Ltd -- http://www.catalyst.net.nz Level 2, 150-154 Willis St, PO Box 11-053, Wellington, NZ Mob: +64 272555332, DDI: +64 4 9167203, Office: +64 4 4992267
Re: repost: [PATCH] SMPP: data_coding (deliver_sm) votes?!
Oded Arbel wrote: This patch was originated from m-Wise code, so I'm not sure I'm allowed to vote on it.. but if for nothing else, at least it introduces iconv into Kannel - which I think is mighty useful. except that, all I can say is works for me(tm). ok, let's interpret this as +1 :) My main concern is what the developers thing of having another hard-wired dependency to third-party software, iconv in this case. Stipe [EMAIL PROTECTED] --- Wapme Systems AG Vogelsanger Weg 80 40470 Düsseldorf Tel: +49-211-74845-0 Fax: +49-211-74845-299 E-Mail: [EMAIL PROTECTED] Internet: http://www.wapme-systems.de --- wapme.net - wherever you are
Re: repost: [PATCH] SMPP: data_coding (deliver_sm) votes?!
On Mittwoch, November 20, 2002, at 03:42 Uhr, Stipe Tolj wrote: Oded Arbel wrote: This patch was originated from m-Wise code, so I'm not sure I'm allowed to vote on it.. but if for nothing else, at least it introduces iconv into Kannel - which I think is mighty useful. except that, all I can say is "works for me(tm)". ok, let's interpret this as +1 :) My main concern is what the developers thing of having another hard-wired dependency to third-party software, iconv in this case. standard libxml bails if iconv is not there. I did run into that problem on earlier MacOS X systems (10.2 and newer is ok) However you can get around this by doing ./configure --without-iconv when you compile libxml. It probably has some replacement function. Now to the stupid question. What is iconv doing at all and why do we need it? Also if libxml has a replacement function for systems which dont have it, can we simply rely on libxml for solving it for us? Andreas Fink Global Networks, Inc. -- Tel: +41-61-333 Fax: +41-61-6932729 Mobile: +41-79-2457333 Global Networks, Inc. Clarastrasse 3, 4058 Basel, Switzerland Web: http://www.global-networks.ch/ [EMAIL PROTECTED] -- Member of the GSM Association
Re: repost: [PATCH] SMPP: data_coding (deliver_sm) votes?!
At 03:55 PM 11/20/02 +0100, you Andreas Fink: On Mittwoch, November 20, 2002, at 03:42 Uhr, Stipe Tolj wrote: Oded Arbel wrote: This patch was originated from m-Wise code, so I'm not sure I'm allowed to vote on it.. but if for nothing else, at least it introduces iconv into Kannel - which I think is mighty useful. except that, all I can say is works for me(tm). ok, let's interpret this as +1 :) My main concern is what the developers thing of having another hard-wired dependency to third-party software, iconv in this case. standard libxml bails if iconv is not there. I did run into that problem on earlier MacOS X systems (10.2 and newer is ok) However you can get around this by doing ./configure --without-iconv when you compile libxml. It probably has some replacement function. Now to the stupid question. What is iconv doing at all and why do we need it? Also if libxml has a replacement function for systems which dont have it, can we simply rely on libxml for solving it for us? I dont think so.. We ended up using it for a proprietary SMSC we connect to. Libxml's replacement functions seem only to handle character encodings needed for XML. Libiconv is also pretty std on most systems I have worked with. Iconv supports many charsets, so another advantage of using it is we have support for charsets that are not yet required or have not been encountered by the list yet. I vote +1 for rolling this patch. We allready use it and it works for us, and its a real pain to keep our source base up to date when there are these differences. Nisan
RE: repost: [PATCH] SMPP: data_coding (deliver_sm) votes?!
My main concern is what the developers thing of having another hard-wired dependency to third-party software, iconv in this case. Yep, that wouldbe a problem. though current implementation means that if iconv isn't there, it will simply not encode- it won't break. standard libxml bails if iconv is not there. I did run into that problem on earlier MacOS X systems (10.2 and newer is ok) However you can get around this by doing ./configure --without-iconv when you compile libxml. It probably has some replacement function. Now to the stupid question. What is iconv doing at all and why do we need it? Also if libxml has a replacement function for systems which dont have it, can we simply rely on libxml for solving it for us? iconv is GNU's character conversion library and it supports every imaginable encoding - unlike libXML who's interface only exposes a very llimited set of character encodings, and its use is also far from being trivial. --Oded Arbelm-Wise mobile solutions[EMAIL PROTECTED] +972-9-9581711 (116)+972-67-340014 ::..People are flexible enough to make any theory look good for a while. -- Jaron Lanier, in his Karma Vertigo essay
Re: repost: [PATCH] SMPP: data_coding (deliver_sm) votes?!
On Mittwoch, November 20, 2002, at 06:52 Uhr, Oded Arbel wrote: My main concern is what the developers thing of having another hard-wired dependency to third-party software, iconv in this case. Yep, that would be a problem. though current implementation means that if iconv isn't there, it will simply not encode- it won't break. standard libxml bails if iconv is not there. I did run into that problem on earlier MacOS X systems (10.2 and newer is ok) However you can get around this by doing ./configure --without-iconv when you compile libxml. It probably has some replacement function. Now to the stupid question. What is iconv doing at all and why do we need it? Also if libxml has a replacement function for systems which dont have it, can we simply rely on libxml for solving it for us? iconv is GNU's character conversion library and it supports every imaginable encoding - unlike libXML who's interface only exposes a very llimited set of character encodings, and its use is also far from being trivial. I would go for a ./configure --with-iconv where the default is without. So people who need it can use it and others who dont, wont. Andreas Fink Global Networks, Inc. -- Tel: +41-61-333 Fax: +41-61-6932729 Mobile: +41-79-2457333 Global Networks, Inc. Clarastrasse 3, 4058 Basel, Switzerland Web: http://www.global-networks.ch/ [EMAIL PROTECTED] -- Member of the GSM Association
RE: repost: [PATCH] SMPP: data_coding (deliver_sm) votes?!
I would go for a ./configure --with-iconv where the default is without. So people who need it can use it and others who dont, wont. After the patch, configure will automaticly detect if iconv exist and will use it. if it does not exist, the charset_convert function will still be defined, but with empty body, which behaves the same except that the text will not be encoded. Its possible to strap some "iconv replacement" logic instead of the empty body so that if iconv exists it will be used, otherwise - the replacement logic. --Oded Arbelm-Wise mobile solutions[EMAIL PROTECTED] +972-9-9581711 (116)+972-67-340014 ::..malpractice, n.:The reason surgeons wear masks.
Re: repost: [PATCH] SMPP: data_coding (deliver_sm) votes?!
I vote +1 for rolling this patch. We allready use it and it works for us, and its a real pain to keep our source base up to date when there are these differences. Sync'ing private cvs trees is not our main intention. Basically we at Wapme do this in some sense too, but we always focus that the officual cvs branch keeps sementically intact. (just to my 2ct in regards to private cvs sync'ing). Ok, if iconv is anyway used in libxml2 (by default?!) then I'd vote +0 for this too and I'd like to hear some more votes from the developers before dropping this in. At least someone may veto to add this, if so, please argument on the vetos. Stipe [EMAIL PROTECTED] --- Wapme Systems AG Vogelsanger Weg 80 40470 Düsseldorf Tel: +49-211-74845-0 Fax: +49-211-74845-299 E-Mail: [EMAIL PROTECTED] Internet: http://www.wapme-systems.de --- wapme.net - wherever you are
Re: repost: [PATCH] SMPP: data_coding (deliver_sm) votes?!
the patch tests on the header files, not on the library itself. BTW, the patch does not work when linking the executables for me, because configure has not added any lib dependecy statements for ld. Ok, I'd propose the following behaviour: configure tests on iconv presence. If available sets #define HAVE_ICONV in config.h and the iconv routines are used directly in the code. If not available use the old behaviour. What about that approach? Stipe [EMAIL PROTECTED] --- Wapme Systems AG Vogelsanger Weg 80 40470 Düsseldorf Tel: +49-211-74845-0 Fax: +49-211-74845-299 E-Mail: [EMAIL PROTECTED] Internet: http://www.wapme-systems.de --- wapme.net - wherever you are
