AW: AW: AW: Why do I still need MacToISO, when working with UTF-8?

[Tiemo Hollmann TB via use-livecode]

Yep.
And no, I didn't tested textEncode(myFile,"CP1252")
Tiemo



-Ursprüngliche Nachricht-
Von: use-livecode [mailto:use-livecode-boun...@lists.runrev.com] Im Auftrag
von Kay C Lan via use-livecode
Gesendet: Mittwoch, 18. Januar 2017 05:36
An: How to use LiveCode <use-livecode@lists.runrev.com>
Cc: Kay C Lan <lan.kc.macm...@gmail.com>
Betreff: Re: AW: AW: Why do I still need MacToISO, when working with UTF-8?

On Tue, Jan 17, 2017 at 1:24 AM, Mark Waddingham via use-livecode
<use-livecode@lists.runrev.com> wrote:
>
> However, the 'endpoints' (i.e. where the developer can 'see' encoded 
> text output - e.g. when writing to a file, or encoding for a URL) had 
> to remain as before otherwise all existing applications using anything 
> other than ASCII text would have broken when moving from 6.7 -> 7.0.
>
But isn't that the point of Tiemo's confusion - his scripts broke when
moving to 7.0! Prior to 7.0 he didn't have to do anything, it all 'just
worked'. When he moved to 7.0 where 'unicode' was suppose to 'just work' on
all platforms, he's used textEncode/textDecode to/from
UTF8 and it's not working for him (on Mac), instead he's found macToISO
(MacRoman to Latin 1) is working for him, which seems to be a step
backwards.

There must be something more hidden in his scripts or PHP.

I wonder if he replaced macToISO(myFile) with
textEncode(myFile,"CP1252") he'd get the same result. If so, it may suggest
that everything is expecting Latin 1, not unicode.

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode


___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: AW: AW: Why do I still need MacToISO, when working with UTF-8?

[Kay C Lan via use-livecode]

On Tue, Jan 17, 2017 at 1:24 AM, Mark Waddingham via use-livecode
 wrote:
>
> However, the 'endpoints' (i.e. where the developer can 'see' encoded text
> output - e.g. when writing to a file, or encoding for a URL) had to remain
> as before otherwise all existing applications using anything other than
> ASCII text would have broken when moving from 6.7 -> 7.0.
>
But isn't that the point of Tiemo's confusion - his scripts broke when
moving to 7.0! Prior to 7.0 he didn't have to do anything, it all
'just worked'. When he moved to 7.0 where 'unicode' was suppose to
'just work' on all platforms, he's used textEncode/textDecode to/from
UTF8 and it's not working for him (on Mac), instead he's found
macToISO (MacRoman to Latin 1) is working for him, which seems to be a
step backwards.

There must be something more hidden in his scripts or PHP.

I wonder if he replaced macToISO(myFile) with
textEncode(myFile,"CP1252") he'd get the same result. If so, it may
suggest that everything is expecting Latin 1, not unicode.

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

AW: Why do I still need MacToISO, when working with UTF-8?

[Tiemo Hollmann TB via use-livecode]

This is what I originally tested first (see my original post), but it didn't
worked. The Umlaute (coming from a Mac client) entered the mySQL db
corrupted. I don't know, if I could configure anything in my PHP or my db
different to make this solution work. But since everything worked always
fine with a windows client, I didn't wanted to change anything in my PHP or
MySQL and so the MacToISO() conversion on a Mac client is a good solution
for my case.
Tiemo


-Ursprüngliche Nachricht-
Von: use-livecode [mailto:use-livecode-boun...@lists.runrev.com] Im Auftrag
von Mark Waddingham via use-livecode
Gesendet: Montag, 16. Januar 2017 18:30
An: How to use LiveCode <use-livecode@lists.runrev.com>
Cc: Mark Waddingham <m...@livecode.com>
Betreff: Re: Why do I still need MacToISO, when working with UTF-8?

Hi Matthias,

On 2017-01-16 18:25, Matthias Rebbe via use-livecode wrote:
> It would have been nice if you had also put some sample code for how 
> to UTF8 encode the string.
> That would have made your explanations complete. ;)

Sure - here is how I'd slightly adjust Tiemo's code:

*put fld "name" into myName*
-- ...
*open file myFile for binary write*
*write textEncode(myName, "utf8") to file myFile* *close file myFile*
-- ...
*open file myFile for binary read*
*read from file myFile until EOF*
*close file myFile*
*put textDecode(it, "utf8") into myName*
-- ...
*put URL ("http://myUser:myPW@myURL; & "mySQL.php?" &
URLEncode(textEncode(theName, "utf8"))) into rslt*
-- mySQL.php writes to a MySQL db, where theName column is encoded as
"utf8_general_ci"

The missing piece here is PHP configuration on the other side. I'm assuming
that PHP is doing the following:
1) URLDecode the part after '?' into bytes
2) Interpret the bytes as Latin-1 encoded text
3) Passing the text string to the appropriate MySQL function
4) The MySQL function is converting the text string to UTF-8

This may or may not be the case. If 'theName' is encoded as UTF8 before
being URLEncode, all that needs to be checked is that the PHP (on the other
end) is decoding it into a string as UTF-8 before passing it to MySQL.

Warmest egards,

Mark.

--
Mark Waddingham ~ m...@livecode.com ~ http://www.livecode.com/
LiveCode: Everyone can create apps

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode


___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: AW: AW: Why do I still need MacToISO, when working with UTF-8?

[Sébastien Nouat via use-livecode]

Hi Tiemo,

As an additional note (if you don't absolutely need to write binary) it 
is also possible to use the syntax


/open file  for "utf8" text read/

or

/open file  for "utf8" text write/


in which cases the engine takes care of encoding/decoding the string 
using UTF-8 encoding.


Then, when calling

/write  to file /

/string/ can be a LiveCode string with Unicode characters, and will be 
written in UTF-8, and


/read from file  for 1 lines

/will set /it/ to a string with the decoded UTF-8 string.


Regards,

Sebastien

On 16/01/2017 17:14, Tiemo Hollmann TB via use-livecode wrote:

Hi Mark,
thank you for taking your time and clarifying. I wasn't aware that the
internal format on a Mac client is MacRoman. I thought it would be a
"neutral" UTF-8 format.
Thanks
Tiemo


-Ursprüngliche Nachricht-
Von: use-livecode [mailto:use-livecode-boun...@lists.runrev.com] Im Auftrag
von Mark Waddingham via use-livecode
Gesendet: Montag, 16. Januar 2017 17:42
An: How to use LiveCode <use-livecode@lists.runrev.com>
Cc: Mark Waddingham <m...@livecode.com>
Betreff: Re: AW: Why do I still need MacToISO, when working with UTF-8?

Hi Tiemo,

Okay so, I'm assuming that all this code is running on the Mac client...


*put fld "name" into myName*

At this point myName contains a (text) string - thus encoding issues don't
exist (you should think of text strings in memory as being stored in an
'encoding neutral' format).


*open file myFile for binary write*
*write myName to file myFile*
*close file myFile*

This piece of code will open a file on disk in the native encoding of the
platform - so MacRoman. It will convert this from the internal encoding to
MacRoman on writing. Thus your text file will be a MacRoman encoded text
file.


*open file myFile for binary read*
*read from file myFile until EOF*
*close file myFile*
*put it into myName*

This piece of code will read from a file on disk and assume that it is in
the native encoding of the platform - so, in this case, MacRoman. It will
convert the content of the file from that to the internal encoding.

Up to this point - because you saved and loaded the file on the same
platform the content of myName should be as you expect -- unchanged.


*if the platform is "MacOS" then put macToISO(theName) into theName*

When run on Mac this line will execute and do the following:

 1) Convert theName to a binary string - this uses the native platform
encoding (MacRoman)
 2) Map each byte from the MacRoman code index to the ISO Latin-1 code
index

This essentially converts theName from a text string to a binary string
encoded in Latin-1.


*put URL ("http://myUser:myPW@myURL; & "mySQL.php?" &
URLEncode(theName))
into rslt*

This line constructs the URL - it is making the assumption that PHP (at the
other end) will interpret the bytes after the '?' as representing
Latin-1 encoded text.


Without macToISO on a Mac client theName enters corrupted in the mySQL
db

This is most likely because PHP is defaulting to 8859-1 or Latin-1 as the
encoding used in URLEncoded fields in a URL. If you don't do MacToIso, then
you will be passing up MacRoman encoded text (URLencoded) to PHP, which can
happily be decoded as Latin-1 or 8859-1 (Latin-1 is a superset of 8859-1),
but with some chars (such as accented letters) in different places.

What you need to do here is explicitly UTF8 encode theName before passing it
to URLEncode, then explicitly decode it as UTF8 on the PHP side (or set a
property in PHP which changes the default assumption about URLs - I
apologise for not being more accurate here, my knowledge of PHP is a little
stale these days!).

Warmest Regards,

Mark.

--
Mark Waddingham ~ m...@livecode.com ~ http://www.livecode.com/
LiveCode: Everyone can create apps

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode


___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode


___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Why do I still need MacToISO, when working with UTF-8?

[Matthias Rebbe via use-livecode]

Mark,

thanks again for your explanations.

That explains some strange things here in the past… ;)

Matthias

Matthias Rebbe
Bramkampsieke 13
32312 Lübbecke
Tel +49 5741 31
+49 160 5504462
Fax: +49 5741 310002
eMail: matth...@m-r-d.de 

BR5 Konverter - BR5 -> MP3 
> Am 16.01.2017 um 18:45 schrieb Mark Waddingham via use-livecode 
> >:
> 
>>> Am 16.01.2017 um 18:30 schrieb Mark Waddingham via use-livecode 
>>>  
>>> >> >>:
>>> Sure - here is how I'd slightly adjust Tiemo's code:
>>> *put fld "name" into myName*
>>> -- ...
>>> *open file myFile for binary write*
>>> *write textEncode(myName, "utf8") to file myFile*
>>> *close file myFile*
>>> -- ...
>>> *open file myFile for binary read*
>>> *read from file myFile until EOF*
>>> *close file myFile*
>>> *put textDecode(it, "utf8") into myName*
>> I always thought, that binary reading a text file would result into a
>> string with the same encoding and  line endings.
> 
> When you read a file in binary mode, what you get is binary data *not* text - 
> i.e. it is just a sequence of bytes. The engine cannot tell by just looking 
> at the bytes what it could be therefore you have to explicitly convert it to 
> something - in this case we convert the sequence of bytes to text by 
> interpreting the bytes as UTF-8.
> 
> One of the biggest changes from 6 to 7 is that binary strings and text 
> strings are no longer the same thing.
> 
> Prior to 7, the engine didn't really 'know' anything about Unicode - the 
> field did to a certain degree, but nothing else - and it assumed that binary 
> strings and text strings were the same thing. Indeed, on Mac the engine would 
> assume that a binary string could be treated as a MacRoman encoded string (as 
> MacRoman is one byte, one char); and on Windows/Linux it would assume that a 
> binary string could be treated as a Latin-1 encoded string (also a one byte, 
> one char encoding).
> 
> This equivalence has been retained in 7 from 6 - which is why stacks written 
> in 6 work exactly the same as they do in 7. Specifically, there is an 
> implicit auto conversion between binary strings and text strings using the 
> platform encoding:
> 
>put  into tVar
>put "foobar" after tVar
> 
> In the second line here, the engine will first convert tVar to a text string 
> (assuming MacRoman encoding on Mac) then append "foobar".
> 
>> So when i binary read UTF8 files  i still have to textDecode it to UTF8?
> 
> Yes - because if you read something as binary, then it is just that - binary 
> - it has no structure and is just a sequence of bytes.
> 
> A perhaps more obviously example is that you have to explicitly decompress 
> data which has been compress'd and explicitly arrayDecode data which has been 
> arrayEncode'd. When it is just data, the engine doesn't know what it could be 
> so the code processing it has to explicitly specify a conversion.
> 
> Warmest Regards,
> 
> Mark.
> 
> -- 
> Mark Waddingham ~ m...@livecode.com  ~ 
> http://www.livecode.com/ 
> LiveCode: Everyone can create apps
> 
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com 
> Please visit this url to subscribe, unsubscribe and manage your subscription 
> preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Why do I still need MacToISO, when working with UTF-8?

[Mark Waddingham via use-livecode]

Am 16.01.2017 um 18:30 schrieb Mark Waddingham via use-livecode 
>:



Sure - here is how I'd slightly adjust Tiemo's code:

*put fld "name" into myName*
-- ...
*open file myFile for binary write*
*write textEncode(myName, "utf8") to file myFile*
*close file myFile*
-- ...
*open file myFile for binary read*
*read from file myFile until EOF*
*close file myFile*
*put textDecode(it, "utf8") into myName*


I always thought, that binary reading a text file would result into a
string with the same encoding and  line endings.


When you read a file in binary mode, what you get is binary data *not* 
text - i.e. it is just a sequence of bytes. The engine cannot tell by 
just looking at the bytes what it could be therefore you have to 
explicitly convert it to something - in this case we convert the 
sequence of bytes to text by interpreting the bytes as UTF-8.


One of the biggest changes from 6 to 7 is that binary strings and text 
strings are no longer the same thing.


Prior to 7, the engine didn't really 'know' anything about Unicode - the 
field did to a certain degree, but nothing else - and it assumed that 
binary strings and text strings were the same thing. Indeed, on Mac the 
engine would assume that a binary string could be treated as a MacRoman 
encoded string (as MacRoman is one byte, one char); and on Windows/Linux 
it would assume that a binary string could be treated as a Latin-1 
encoded string (also a one byte, one char encoding).


This equivalence has been retained in 7 from 6 - which is why stacks 
written in 6 work exactly the same as they do in 7. Specifically, there 
is an implicit auto conversion between binary strings and text strings 
using the platform encoding:


put  into tVar
put "foobar" after tVar

In the second line here, the engine will first convert tVar to a text 
string (assuming MacRoman encoding on Mac) then append "foobar".


So when i binary read UTF8 files  i still have to textDecode it to 
UTF8?


Yes - because if you read something as binary, then it is just that - 
binary - it has no structure and is just a sequence of bytes.


A perhaps more obviously example is that you have to explicitly 
decompress data which has been compress'd and explicitly arrayDecode 
data which has been arrayEncode'd. When it is just data, the engine 
doesn't know what it could be so the code processing it has to 
explicitly specify a conversion.


Warmest Regards,

Mark.

--
Mark Waddingham ~ m...@livecode.com ~ http://www.livecode.com/
LiveCode: Everyone can create apps

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Why do I still need MacToISO, when working with UTF-8?

[Matthias Rebbe via use-livecode]

Hi Mark,

thanks for that.

> Am 16.01.2017 um 18:30 schrieb Mark Waddingham via use-livecode 
> >:
> 
> 
> Sure - here is how I'd slightly adjust Tiemo's code:
> 
> *put fld "name" into myName*
> -- ...
> *open file myFile for binary write*
> *write textEncode(myName, "utf8") to file myFile*
> *close file myFile*
> -- ...
> *open file myFile for binary read*
> *read from file myFile until EOF*
> *close file myFile*
> *put textDecode(it, "utf8") into myName*

I always thought, that binary reading a text file would result into a string 
with the same encoding and  line endings.
So when i binary read UTF8 files  i still have to textDecode it to UTF8?





> -- ...
> *put URL ("http://myUser:myPW@myURL " & 
> "mySQL.php?" & URLEncode(textEncode(theName, "utf8")))
> into rslt*
> -- mySQL.php writes to a MySQL db, where theName column is encoded as
> "utf8_general_ci"
> 
> The missing piece here is PHP configuration on the other side. I'm assuming 
> that PHP is doing the following:
>   1) URLDecode the part after '?' into bytes
>   2) Interpret the bytes as Latin-1 encoded text
>   3) Passing the text string to the appropriate MySQL function
>   4) The MySQL function is converting the text string to UTF-8
> 
> This may or may not be the case. If 'theName' is encoded as UTF8 before being 
> URLEncode, all that needs to be checked is that the PHP (on the other end) is 
> decoding it into a string as UTF-8 before passing it to MySQL.
> 
> Warmest egards,
> 
> Mark.
> 
> -- 
> Mark Waddingham ~ m...@livecode.com  ~ 
> http://www.livecode.com/ 
> LiveCode: Everyone can create apps
> 
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com 
> Please visit this url to subscribe, unsubscribe and manage your subscription 
> preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Why do I still need MacToISO, when working with UTF-8?

[Mark Waddingham via use-livecode]

Hi Matthias,

On 2017-01-16 18:25, Matthias Rebbe via use-livecode wrote:

It would have been nice if you had also put some sample code for how
to UTF8 encode the string.
That would have made your explanations complete. ;)


Sure - here is how I'd slightly adjust Tiemo's code:

*put fld "name" into myName*
-- ...
*open file myFile for binary write*
*write textEncode(myName, "utf8") to file myFile*
*close file myFile*
-- ...
*open file myFile for binary read*
*read from file myFile until EOF*
*close file myFile*
*put textDecode(it, "utf8") into myName*
-- ...
*put URL ("http://myUser:myPW@myURL; & "mySQL.php?" & 
URLEncode(textEncode(theName, "utf8")))

into rslt*
-- mySQL.php writes to a MySQL db, where theName column is encoded as
"utf8_general_ci"

The missing piece here is PHP configuration on the other side. I'm 
assuming that PHP is doing the following:

   1) URLDecode the part after '?' into bytes
   2) Interpret the bytes as Latin-1 encoded text
   3) Passing the text string to the appropriate MySQL function
   4) The MySQL function is converting the text string to UTF-8

This may or may not be the case. If 'theName' is encoded as UTF8 before 
being URLEncode, all that needs to be checked is that the PHP (on the 
other end) is decoding it into a string as UTF-8 before passing it to 
MySQL.


Warmest egards,

Mark.

--
Mark Waddingham ~ m...@livecode.com ~ http://www.livecode.com/
LiveCode: Everyone can create apps

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Why do I still need MacToISO, when working with UTF-8?

[Matthias Rebbe via use-livecode]

Hi Mark,

although i am not the original poster i really appreciate you explanation. 

The only thing what was missing …

It would have been nice if you had also put some sample code for how to UTF8 
encode the string.
That would have made your explanations complete. ;)

I am pretty sure there are many on the list who are struggling with that 
econding part. Or am i the only one?


Regards,
Matthias




> Am 16.01.2017 um 17:41 schrieb Mark Waddingham via use-livecode 
> >:
> 
> 
> 
> What you need to do here is explicitly UTF8 encode theName before passing it 
> to URLEncode, then explicitly decode it as UTF8 on the PHP side (or set a 
> property in PHP which changes the default assumption about URLs - I apologise 
> for not being more accurate here, my knowledge of PHP is a little stale these 
> days!).
> 
> Warmest Regards,
> 
> Mark.
> 
> -- 
> Mark Waddingham ~ m...@livecode.com  ~ 
> http://www.livecode.com/ 
> LiveCode: Everyone can create apps
> 
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com 
> Please visit this url to subscribe, unsubscribe and manage your subscription 
> preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: AW: AW: Why do I still need MacToISO, when working with UTF-8?

[Mark Waddingham via use-livecode]

Hi Tiemo,


thank you for taking your time and clarifying. I wasn't aware that the
internal format on a Mac client is MacRoman. I thought it would be a
"neutral" UTF-8 format.


Internally, the engine uses either MacRoman/ISO-Latin1 *or* UTF-16 
depending on platform and what the string contains.


However, the 'endpoints' (i.e. where the developer can 'see' encoded 
text output - e.g. when writing to a file, or encoding for a URL) had to 
remain as before otherwise all existing applications using anything 
other than ASCII text would have broken when moving from 6.7 -> 7.0.


You can use the 'utf8' keyword to open utf-8 encoded files; however, you 
have to deal with urlEncode manually (which isn't necessarily a bad 
thing, since your server scripts determines what the URL Encoded bytes 
mean after the '?' - NOT LiveCode).


Warmest Regards,

Mark.

--
Mark Waddingham ~ m...@livecode.com ~ http://www.livecode.com/
LiveCode: Everyone can create apps

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

AW: AW: Why do I still need MacToISO, when working with UTF-8?

[Tiemo Hollmann TB via use-livecode]

Hi Mark,
thank you for taking your time and clarifying. I wasn't aware that the
internal format on a Mac client is MacRoman. I thought it would be a
"neutral" UTF-8 format.
Thanks
Tiemo


-Ursprüngliche Nachricht-
Von: use-livecode [mailto:use-livecode-boun...@lists.runrev.com] Im Auftrag
von Mark Waddingham via use-livecode
Gesendet: Montag, 16. Januar 2017 17:42
An: How to use LiveCode <use-livecode@lists.runrev.com>
Cc: Mark Waddingham <m...@livecode.com>
Betreff: Re: AW: Why do I still need MacToISO, when working with UTF-8?

Hi Tiemo,

Okay so, I'm assuming that all this code is running on the Mac client...

> *put fld "name" into myName*

At this point myName contains a (text) string - thus encoding issues don't
exist (you should think of text strings in memory as being stored in an
'encoding neutral' format).

> *open file myFile for binary write*
> *write myName to file myFile*
> *close file myFile*

This piece of code will open a file on disk in the native encoding of the
platform - so MacRoman. It will convert this from the internal encoding to
MacRoman on writing. Thus your text file will be a MacRoman encoded text
file.

> *open file myFile for binary read*
> *read from file myFile until EOF*
> *close file myFile*
> *put it into myName*

This piece of code will read from a file on disk and assume that it is in
the native encoding of the platform - so, in this case, MacRoman. It will
convert the content of the file from that to the internal encoding.

Up to this point - because you saved and loaded the file on the same
platform the content of myName should be as you expect -- unchanged.

> *if the platform is "MacOS" then put macToISO(theName) into theName*

When run on Mac this line will execute and do the following:

1) Convert theName to a binary string - this uses the native platform
encoding (MacRoman)
2) Map each byte from the MacRoman code index to the ISO Latin-1 code
index

This essentially converts theName from a text string to a binary string
encoded in Latin-1.

> *put URL ("http://myUser:myPW@myURL; & "mySQL.php?" &
> URLEncode(theName))
> into rslt*

This line constructs the URL - it is making the assumption that PHP (at the
other end) will interpret the bytes after the '?' as representing
Latin-1 encoded text.

> Without macToISO on a Mac client theName enters corrupted in the mySQL 
> db

This is most likely because PHP is defaulting to 8859-1 or Latin-1 as the
encoding used in URLEncoded fields in a URL. If you don't do MacToIso, then
you will be passing up MacRoman encoded text (URLencoded) to PHP, which can
happily be decoded as Latin-1 or 8859-1 (Latin-1 is a superset of 8859-1),
but with some chars (such as accented letters) in different places.

What you need to do here is explicitly UTF8 encode theName before passing it
to URLEncode, then explicitly decode it as UTF8 on the PHP side (or set a
property in PHP which changes the default assumption about URLs - I
apologise for not being more accurate here, my knowledge of PHP is a little
stale these days!).

Warmest Regards,

Mark.

--
Mark Waddingham ~ m...@livecode.com ~ http://www.livecode.com/
LiveCode: Everyone can create apps

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode


___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: AW: Why do I still need MacToISO, when working with UTF-8?

[Mark Waddingham via use-livecode]

Hi Tiemo,

Okay so, I'm assuming that all this code is running on the Mac client...


*put fld "name" into myName*


At this point myName contains a (text) string - thus encoding issues 
don't exist (you should think of text strings in memory as being stored 
in an 'encoding neutral' format).



*open file myFile for binary write*
*write myName to file myFile*
*close file myFile*


This piece of code will open a file on disk in the native encoding of 
the platform - so MacRoman. It will convert this from the internal 
encoding to MacRoman on writing. Thus your text file will be a MacRoman 
encoded text file.



*open file myFile for binary read*
*read from file myFile until EOF*
*close file myFile*
*put it into myName*


This piece of code will read from a file on disk and assume that it is 
in the native encoding of the platform - so, in this case, MacRoman. It 
will convert the content of the file from that to the internal encoding.


Up to this point - because you saved and loaded the file on the same 
platform the content of myName should be as you expect -- unchanged.



*if the platform is "MacOS" then put macToISO(theName) into theName*


When run on Mac this line will execute and do the following:

   1) Convert theName to a binary string - this uses the native platform 
encoding (MacRoman)
   2) Map each byte from the MacRoman code index to the ISO Latin-1 code 
index


This essentially converts theName from a text string to a binary string 
encoded in Latin-1.


*put URL ("http://myUser:myPW@myURL; & "mySQL.php?" & 
URLEncode(theName))

into rslt*


This line constructs the URL - it is making the assumption that PHP (at 
the other end) will interpret the bytes after the '?' as representing 
Latin-1 encoded text.


Without macToISO on a Mac client theName enters corrupted in the mySQL 
db


This is most likely because PHP is defaulting to 8859-1 or Latin-1 as 
the encoding used in URLEncoded fields in a URL. If you don't do 
MacToIso, then you will be passing up MacRoman encoded text (URLencoded) 
to PHP, which can happily be decoded as Latin-1 or 8859-1 (Latin-1 is a 
superset of 8859-1), but with some chars (such as accented letters) in 
different places.


What you need to do here is explicitly UTF8 encode theName before 
passing it to URLEncode, then explicitly decode it as UTF8 on the PHP 
side (or set a property in PHP which changes the default assumption 
about URLs - I apologise for not being more accurate here, my knowledge 
of PHP is a little stale these days!).


Warmest Regards,

Mark.

--
Mark Waddingham ~ m...@livecode.com ~ http://www.livecode.com/
LiveCode: Everyone can create apps

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

AW: Why do I still need MacToISO, when working with UTF-8?

[Tiemo Hollmann TB via use-livecode]

Hi Mark,

as a simplified pseudo code it looks like this:

*put fld "name" into myName*
-- ...
*open file myFile for binary write*
*write myName to file myFile*
*close file myFile*
-- ...
*open file myFile for binary read*
*read from file myFile until EOF*
*close file myFile*
*put it into myName*
-- ...
*if the platform is "MacOS" then put macToISO(theName) into theName*
*put URL ("http://myUser:myPW@myURL; & "mySQL.php?" & URLEncode(theName))
into rslt*
-- mySQL.php writes to a MySQL db, where theName column is encoded as
"utf8_general_ci"
-- ...
Without macToISO on a Mac client theName enters corrupted in the mySQL db

Tiemo


-Ursprüngliche Nachricht-
Von: use-livecode [mailto:use-livecode-boun...@lists.runrev.com] Im Auftrag
von Mark Waddingham via use-livecode
Gesendet: Montag, 16. Januar 2017 14:45
An: How to use LiveCode <use-livecode@lists.runrev.com>
Cc: Mark Waddingham <m...@livecode.com>
Betreff: Re: Why do I still need MacToISO, when working with UTF-8?

Hi Tiemo,

On 2017-01-16 11:57, Tiemo Hollmann TB via use-livecode wrote:
> Now my German Umlaute don't get corrupted in the MySQL db and 
> everything is fine, but I would like to understand the technical 
> background. Why do I still need MacToISO() in LC 8 on a Mac and even 
> worse, I didn't needed it in LC 6 in the same program. What am I 
> missing here?

Can you explain (with code examples if possible) what you are calling
MacToIso() on, and how you are using its output?

It isn't entirely clear exactly what steps you are taking in your previous
email and it sounds like the problem is at the point of LC's communication
with PHP, rather than within LC.

Thanks in advance!

Mark.

--
Mark Waddingham ~ m...@livecode.com ~ http://www.livecode.com/
LiveCode: Everyone can create apps

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode


___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Why do I still need MacToISO, when working with UTF-8?

[Mark Waddingham via use-livecode]

Hi Tiemo,

On 2017-01-16 11:57, Tiemo Hollmann TB via use-livecode wrote:
Now my German Umlaute don't get corrupted in the MySQL db and 
everything is

fine, but I would like to understand the technical background. Why do I
still need MacToISO() in LC 8 on a Mac and even worse, I didn't needed 
it in

LC 6 in the same program. What am I missing here?


Can you explain (with code examples if possible) what you are calling 
MacToIso() on, and how you are using its output?


It isn't entirely clear exactly what steps you are taking in your 
previous email and it sounds like the problem is at the point of LC's 
communication with PHP, rather than within LC.


Thanks in advance!

Mark.

--
Mark Waddingham ~ m...@livecode.com ~ http://www.livecode.com/
LiveCode: Everyone can create apps

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Why do I still need MacToISO, when working with UTF-8?

[Tiemo Hollmann TB via use-livecode]

Hello,

Probably my knowledge about code sets is too limited. My thinking was, that
UTF-8 includes the same characters above ASCII 128 independent of the
platform and makes data transfer between different platform compatible
without conversion of a char set. Migrating a LC 6 program to LC 8 I just
recently needed an extra MacToISO(), before passing a text from my UTF-8 LC
8  program via PHP to a UTF-8 MySQL db on a Unix server.

 

Now my German Umlaute don't get corrupted in the MySQL db and everything is
fine, but I would like to understand the technical background. Why do I
still need MacToISO() in LC 8 on a Mac and even worse, I didn't needed it in
LC 6 in the same program. What am I missing here?

 

Thanks for rerailing me

Tiemo

 

 

 

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode