Re: UTF-8 decoding error
On Mon, 2006-10-09 at 13:55 +0200, Christian Maeder wrote: > Duncan Coutts schrieb: > > On Fri, 2006-09-22 at 17:19 +0200, Christian Maeder wrote: > > >>> A simple script for the pgmF command > > >> The only disadvantage is that the filename in error and warning messages > >> is quite useless: > > > I think you can fix this by pre-pending a {-# LINE #-} pragma in your > > script. Something like: > > May it be that import chasing takes longer now? I noticed quite a gap > before ghc started to compile my 624 modules. Yes, since it has to run the pre-processor before it can look at the imports. Duncan ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: UTF-8 decoding error
Duncan Coutts schrieb: > On Fri, 2006-09-22 at 17:19 +0200, Christian Maeder wrote: >>> A simple script for the pgmF command >> The only disadvantage is that the filename in error and warning messages >> is quite useless: > I think you can fix this by pre-pending a {-# LINE #-} pragma in your > script. Something like: May it be that import chasing takes longer now? I noticed quite a gap before ghc started to compile my 624 modules. Christian ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: UTF-8 decoding error
On Sep 27, 2006, at 6:05 AM, Matthew Pocock wrote: Fortress (sun's possibly-not-vaporware hpc language) supports arbitrary unicode chars in code, and has an escape syntax for commonly used things. I have spent the past week writing Fortress code (which runs in parallel, even). But I'm perhaps a special case. :-) Similarly, proof-general/isabelle supports tex-style escapes for symbols & greek. It seems to me that a pre-processor that turns human- friendly escapes (e.g. \{lambda} rather than some magic number) into unicode and a slightly intelligent IDE (or emacs mode?) would go most of the way to letting us use up-side-down ys and curly as with all the visual beauty and editor niceness that we have now with ascii. In Fortress we spent a *lot* of effort making the "TWiki" syntax as painless as possible for stuff which we planned to use often (for example, -> and => turn into Unicode arrows, and the language syntax is defined in terms of them). One source of both encouragement and frustration is the fact that every unicode code point has an associated description. We support using these descriptions---and various shortenings of them, since they are too verbose for day-to- day use. The frustration is that the names or their shortenings are not necessarily unique. For characters which only occur in strings this is less critical, but a little effort will go a long way. One heuristic we've used is: "if I do a diff on the ASCII representation provided by my version control system, will I be able to read the result?" We of course have a little program which processes an official unicode character table (downloaded from the web) plus some information about our special cases and uses it to generate the appropriate conversion functions. This is important because Unicode is constantly changing (mostly getting bigger). -Jan-Willem Maessen Fortress developer, Haskell hacker Matthew On Wednesday 20 September 2006 21:42, Duncan Coutts wrote: On Wed, 2006-09-20 at 18:14 +0200, Christian Maeder wrote: How can I convince ghc version 6.5.20060919 to accept latin1 characters in literals? I wish to keep source files (containing umlauts in strings) that can be compiled by either ghc-6.4.2 and ghc-6.6. You can use numeric escapes like "\222". Duncan ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users smime.p7s Description: S/MIME cryptographic signature ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: UTF-8 decoding error
Fortress (sun's possibly-not-vaporware hpc language) supports arbitrary unicode chars in code, and has an escape syntax for commonly used things. Similarly, proof-general/isabelle supports tex-style escapes for symbols & greek. It seems to me that a pre-processor that turns human-friendly escapes (e.g. \{lambda} rather than some magic number) into unicode and a slightly intelligent IDE (or emacs mode?) would go most of the way to letting us use up-side-down ys and curly as with all the visual beauty and editor niceness that we have now with ascii. Matthew On Wednesday 20 September 2006 21:42, Duncan Coutts wrote: > On Wed, 2006-09-20 at 18:14 +0200, Christian Maeder wrote: > > How can I convince ghc version 6.5.20060919 to accept latin1 characters > > in literals? > > > > I wish to keep source files (containing umlauts in strings) that can be > > compiled by either ghc-6.4.2 and ghc-6.6. > > You can use numeric escapes like "\222". > > Duncan > > ___ > Glasgow-haskell-users mailing list > Glasgow-haskell-users@haskell.org > http://www.haskell.org/mailman/listinfo/glasgow-haskell-users ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: UTF-8 decoding error
Duncan Coutts schrieb: >> /tmp/ghc5667_0/ghc5667_248.hspp:299:17: > > I think you can fix this by pre-pending a {-# LINE #-} pragma in your > script. Something like: > > #!/bin/sh > ( echo "{-# LINE 1 \"$2\" #-}" ; iconv -f l1 -t utf-8 $2 ) > $3 Yes, thanks again! C. ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: UTF-8 decoding error
On Fri, 2006-09-22 at 17:19 +0200, Christian Maeder wrote: > Christian Maeder schrieb: > > Duncan Coutts schrieb: > >> There is iconv. It could be used as a pre-processor with ghc's -F -pgmF > >> -optF flags. > > > > NB: -F is missing in the Flag reference > > > > A simple script for the pgmF command > > > > #!/bin/sh > > iconv -f l1 -t utf-8 $2 > $3 > > > > worked for me, thanks! > > The only disadvantage is that the filename in error and warning messages > is quite useless: > > [ 15 of 400] Compiling Data.Generics2.Instances ( > syb-generics/Data/Generics2/Instances.hs, > syb-generics/Data/Generics2/Instances.o ) > > /tmp/ghc5667_0/ghc5667_248.hspp:299:17: I think you can fix this by pre-pending a {-# LINE #-} pragma in your script. Something like: #!/bin/sh ( echo "{-# LINE 1 \"$2\" #-}" ; iconv -f l1 -t utf-8 $2 ) > $3 Duncan ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
PS. compiler change, was: UTF-8 decoding error
Christian Maeder schrieb: > [ 15 of 400] Compiling Data.Generics2.Instances ( > syb-generics/Data/Generics2/Instances.hs, > syb-generics/Data/Generics2/Instances.o ) > > /tmp/ghc5667_0/ghc5667_248.hspp:299:17: > Couldn't match expected type `forall a1. (Data ctx a1) => c (t a1)' >against inferred type `c1 (t1 a1)' > Expected type: (forall a2. (Data ctx a2) => c (t a2)) > -> Maybe (c [a]) > Inferred type: c1 (t1 a1) -> Maybe (c1 (t' a1)) > In the expression: gcast1 > In the definition of `dataCast1': dataCast1 _ = gcast1 This particular error is fixed by writing: dataCast1 _ f = gcast1 f (for "dataCast1 _ = gcast1") C. ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: UTF-8 decoding error
Christian Maeder schrieb: > Duncan Coutts schrieb: >> There is iconv. It could be used as a pre-processor with ghc's -F -pgmF >> -optF flags. > > NB: -F is missing in the Flag reference > > A simple script for the pgmF command > > #!/bin/sh > iconv -f l1 -t utf-8 $2 > $3 > > worked for me, thanks! The only disadvantage is that the filename in error and warning messages is quite useless: [ 15 of 400] Compiling Data.Generics2.Instances ( syb-generics/Data/Generics2/Instances.hs, syb-generics/Data/Generics2/Instances.o ) /tmp/ghc5667_0/ghc5667_248.hspp:299:17: Couldn't match expected type `forall a1. (Data ctx a1) => c (t a1)' against inferred type `c1 (t1 a1)' Expected type: (forall a2. (Data ctx a2) => c (t a2)) -> Maybe (c [a]) Inferred type: c1 (t1 a1) -> Maybe (c1 (t' a1)) In the expression: gcast1 In the definition of `dataCast1': dataCast1 _ = gcast1 ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: UTF-8 decoding error
Duncan Coutts schrieb: > There is iconv. It could be used as a pre-processor with ghc's -F -pgmF > -optF flags. NB: -F is missing in the Flag reference A simple script for the pgmF command #!/bin/sh iconv -f l1 -t utf-8 $2 > $3 worked for me, thanks! > Sorry there isn't a better solution at the moment. You could petition > for an {-# ENCODING ISO-8859-1 #-} pragma as mentioned in that ticket. This seems unnecessary now. Christian ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: UTF-8 decoding error
currently haddock correctly translates latin1 chars, ie. äöü to äöü So it would be nice if also ghc-6.6 could remain backward compatible by supporting latin1 sources. Christian Ross Paterson schrieb: > On Thu, Sep 21, 2006 at 10:15:45AM +0200, Christian Maeder wrote: >> How does haddock handle characters in comments? > > Section 3.8.3 of the Haddock manual: > > 3.8.3. Character references > > Although Haskell source files may contain any character from the > Unicode character set, the encoding of these characters as bytes > varies between systems, so that only source files restricted to > the ASCII character set are portable. Other characters may be > specified in character and string literals using Haskell character > escapes. To represent such characters in documentation comments, > Haddock supports SGML-style numeric character references of the > forms D; and H; where D and H are decimal and hexadecimal > numbers denoting a code position in Unicode (or ISO 10646). > For example, the references λ, λ and λ all > represent the lower-case letter lambda. > > Not pretty, but it is portable and not limited to the Latin-1 subset. ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: UTF-8 decoding error
On Thu, 2006-09-21 at 10:15 +0200, Christian Maeder wrote: > Duncan Coutts schrieb: > > On Wed, 2006-09-20 at 18:14 +0200, Christian Maeder wrote: > >> How can I convince ghc version 6.5.20060919 to accept latin1 characters > >> in literals? > >> > >> I wish to keep source files (containing umlauts in strings) that can be > >> compiled by either ghc-6.4.2 and ghc-6.6. > > > > You can use numeric escapes like "\222". > > How about resolving http://cvs.haskell.org/trac/ghc/ticket/690 ? > > I would like to simple add a flag to ghc. How about adding an addional > phase (and extension) to ghc before preprocessing? > > Your solution does not support good readability. Is there a ready tool > that converts my source files as you suggested automatically? How does > haddock handle characters in comments? There is iconv. It could be used as a pre-processor with ghc's -F -pgmF -optF flags. Sorry there isn't a better solution at the moment. You could petition for an {-# ENCODING ISO-8859-1 #-} pragma as mentioned in that ticket. Duncan ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: UTF-8 decoding error
On Thu, Sep 21, 2006 at 10:15:45AM +0200, Christian Maeder wrote: > How does haddock handle characters in comments? Section 3.8.3 of the Haddock manual: 3.8.3. Character references Although Haskell source files may contain any character from the Unicode character set, the encoding of these characters as bytes varies between systems, so that only source files restricted to the ASCII character set are portable. Other characters may be specified in character and string literals using Haskell character escapes. To represent such characters in documentation comments, Haddock supports SGML-style numeric character references of the forms D; and H; where D and H are decimal and hexadecimal numbers denoting a code position in Unicode (or ISO 10646). For example, the references λ, λ and λ all represent the lower-case letter lambda. Not pretty, but it is portable and not limited to the Latin-1 subset. ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: UTF-8 decoding error
Duncan Coutts schrieb: > On Wed, 2006-09-20 at 18:14 +0200, Christian Maeder wrote: >> How can I convince ghc version 6.5.20060919 to accept latin1 characters >> in literals? >> >> I wish to keep source files (containing umlauts in strings) that can be >> compiled by either ghc-6.4.2 and ghc-6.6. > > You can use numeric escapes like "\222". How about resolving http://cvs.haskell.org/trac/ghc/ticket/690 ? I would like to simple add a flag to ghc. How about adding an addional phase (and extension) to ghc before preprocessing? Your solution does not support good readability. Is there a ready tool that converts my source files as you suggested automatically? How does haddock handle characters in comments? I'm sort of stuck currently testing ghc-6.6 RC Christian ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: UTF-8 decoding error
On Wed, 2006-09-20 at 18:14 +0200, Christian Maeder wrote: > How can I convince ghc version 6.5.20060919 to accept latin1 characters > in literals? > > I wish to keep source files (containing umlauts in strings) that can be > compiled by either ghc-6.4.2 and ghc-6.6. You can use numeric escapes like "\222". Duncan ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
UTF-8 decoding error
How can I convince ghc version 6.5.20060919 to accept latin1 characters in literals? I wish to keep source files (containing umlauts in strings) that can be compiled by either ghc-6.4.2 and ghc-6.6. Christian ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: UTF-8 decoding
Marcin 'Qrczak' Kowalczyk wrote: John Meacham <[EMAIL PROTECTED]> writes: Another possibility is quasi-utf8 encoding. where it passes through any invalid utf8 sequences as latin1 characters. in practice, this works very well as interpreting both latin1 and utf8 transparently but is more than somewhat hacky. It would not be reliable. I'm strongly against that: it gives an illusion that Latin1 works, but it breaks in very rare cases. I tend to agree with Marcin here - that doesn't sound like a good solution. Incedentally, we do ignore encoding errors in comments (more by accident than by design, though :-). Cheers, Simon ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: UTF-8 decoding
John Meacham <[EMAIL PROTECTED]> writes: > Another possibility is quasi-utf8 encoding. where it passes through any > invalid utf8 sequences as latin1 characters. in practice, this works > very well as interpreting both latin1 and utf8 transparently but is > more than somewhat hacky. It would not be reliable. I'm strongly against that: it gives an illusion that Latin1 works, but it breaks in very rare cases. -- __("< Marcin Kowalczyk \__/ [EMAIL PROTECTED] ^^ http://qrnik.knm.org.pl/~qrczak/ ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: UTF-8 decoding
On Fri, Feb 10, 2006 at 12:50:57PM +, Simon Marlow wrote: > That's true. I guess what you're saying is that this is a problem for > you, and your life would be easier if we supported Latin-1 as an > encoding for source files again. That's fine - as I mentioned, I only > restricted it to UTF-8 initially because (a) it was easier and (b) I > wanted to see if anyone would be adversely affected. Another possibility is quasi-utf8 encoding. where it passes through any invalid utf8 sequences as latin1 characters. in practice, this works very well as interpreting both latin1 and utf8 transparently but is more than somewhat hacky. perhaps we can have quasi-utf8 be the default, with strict utf8 and latin1 being switches? John -- John Meacham - ⑆repetae.net⑆john⑈ ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: UTF-8 decoding
Christian Maeder wrote: Simon Marlow wrote: I'm not sure I see the problem - the I/O library doesn't do unicode encoding/decoding, it always just takes the low 8 bits of each character, hence truncating Unicode to Latin-1. If you restrict yourself to Latin-1 characters in string literals, then I/O will work as expected (i.e. Latin-1 only). But if ghc-6.5 will expect utf-8 encoded source files all other haskell applications reading or writing haskell files must be adapted as well or am I wrong? That's true. I guess what you're saying is that this is a problem for you, and your life would be easier if we supported Latin-1 as an encoding for source files again. That's fine - as I mentioned, I only restricted it to UTF-8 initially because (a) it was easier and (b) I wanted to see if anyone would be adversely affected. I've now added a ticket for this: http://cvs.haskell.org/trac/ghc/ticket/690 Thanks for the feedback! Cheers, Simon ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
UTF-8 decoding
Simon Marlow wrote: I'm not sure I see the problem - the I/O library doesn't do unicode encoding/decoding, it always just takes the low 8 bits of each character, hence truncating Unicode to Latin-1. If you restrict yourself to Latin-1 characters in string literals, then I/O will work as expected (i.e. Latin-1 only). But if ghc-6.5 will expect utf-8 encoded source files all other haskell applications reading or writing haskell files must be adapted as well or am I wrong? C. ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: UTF-8 decoding error
Christian Maeder wrote: Simon Marlow wrote: Christian Maeder wrote: I'm tempted to replace "ä" bei "\228" in literals. What does haddock do with utf-8 in comments? Will DrIFT -- using read- and writeFile -- still work correctly? The problem I fear is that writeFile does not produce a utf-8 encoded file: writeFile "t.hs" "main = putStrLn \"äöüßÄÖÜ\"" Using "\228\246\252\223\196\214\220" instead of "äöüßÄÖÜ" only avoids conversion to utf-8 of the initial file l1.hs (attached), but the generated file t.hs is a latin-1 file in both cases. Cheers Christian *Main> :l l1.hs Compiling Main ( l1.hs, interpreted ) Ok, modules loaded: Main. *Main> main *Main> :l t.hs Compiling Main ( t.hs, interpreted ) Ok, modules loaded: Main. *Main> main äöüßÄÖÜ I'm not sure I see the problem - the I/O library doesn't do unicode encoding/decoding, it always just takes the low 8 bits of each character, hence truncating Unicode to Latin-1. If you restrict yourself to Latin-1 characters in string literals, then I/O will work as expected (i.e. Latin-1 only). If you need to do I/O in a different encoding, I'm afraid you'll have to code it up yourself right now, or use some other library (there are packed string libraries around that can do I/O in UTF-8, for example, and Bulat's new I/O library does char encodings). Cheers, Simon ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: UTF-8 decoding error
Simon Marlow wrote: Christian Maeder wrote: I'm tempted to replace "ä" bei "\228" in literals. What does haddock do with utf-8 in comments? Will DrIFT -- using read- and writeFile -- still work correctly? The problem I fear is that writeFile does not produce a utf-8 encoded file: writeFile "t.hs" "main = putStrLn \"äöüßÄÖÜ\"" Using "\228\246\252\223\196\214\220" instead of "äöüßÄÖÜ" only avoids conversion to utf-8 of the initial file l1.hs (attached), but the generated file t.hs is a latin-1 file in both cases. Cheers Christian *Main> :l l1.hs Compiling Main ( l1.hs, interpreted ) Ok, modules loaded: Main. *Main> main *Main> :l t.hs Compiling Main ( t.hs, interpreted ) Ok, modules loaded: Main. *Main> main äöüßÄÖÜ main = writeFile "t.hs" "main = putStrLn \"äöüßÄÖÜ\"" ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: UTF-8 decoding error
Christian Maeder wrote: So - do you need Latin-1, or could you use UTF-8? I'm not amused to change the encoding of many haskell source files (particular of those that are not mine). Fair enough, but there will have to be some way to specify the encoding, either via a pragma, command-line option, or the locale. I'm really not sure what is the best choice here. Perhaps all three, with locale being the default, overriden by pragmas and command-line options. The easiest way for us to handle encodings other than UTF-8 is for it to be a new preprocessing step, running 'iconv'. (but what do we do on Windows? bundle iconv? ew.) John - what do you plan to do here? These files can then no longer be compiled by earlier ghcs (though I don't understand, how ghc-6.4.1 recognises the lexical error). I'm tempted to replace "ä" bei "\228" in literals. What does haddock do with utf-8 in comments? Will DrIFT -- using read- and writeFile -- still work correctly? Haddock needs to be updated too. But if GHC implements recoding via iconv, you can use GHC as a preprocesor to recode back to Latin-1; since you have to use GHC as a preprocessor with Haddock anyway, this shouldn't be much harder (of course, if you use non-Latin-1 characters this fails). Eventually, when Haddock runs on top of GHC, the issue will go away :) I don't know about DrIFT. Cheers, Simon ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: UTF-8 decoding error
Simon Marlow wrote: So - do you need Latin-1, or could you use UTF-8? I'm not amused to change the encoding of many haskell source files (particular of those that are not mine). These files can then no longer be compiled by earlier ghcs (though I don't understand, how ghc-6.4.1 recognises the lexical error). I'm tempted to replace "ä" bei "\228" in literals. What does haddock do with utf-8 in comments? Will DrIFT -- using read- and writeFile -- still work correctly? Cheers Christian ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Re: UTF-8 decoding error
Christian Maeder wrote: with ghc-6.5.20060201 I get a "UTF-8 decoding error" for latin1 characters in my string literals. Do I have to change my sources or can I set a certain environment variable? I have [EMAIL PROTECTED] and LC_CTYPE not set (which is ok for hugs) GHC is now expected source files to be UTF-8 only. I really did this as an experiment to see if anyone complained, because it will be more work to implement other encodings. You're the second person to notice this. So - do you need Latin-1, or could you use UTF-8? If you're using emacs, it's pretty easy to default to UTF-8 for haskell source files, BTW. Just add this to your .emacs: (modify-coding-system-alist 'file "\\.l?hs\\'" 'utf-8) Cheers, Simon ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
UTF-8 decoding error
Hi, with ghc-6.5.20060201 I get a "UTF-8 decoding error" for latin1 characters in my string literals. Do I have to change my sources or can I set a certain environment variable? I have [EMAIL PROTECTED] and LC_CTYPE not set (which is ok for hugs) Cheers Christian ___ Glasgow-haskell-users mailing list Glasgow-haskell-users@haskell.org http://www.haskell.org/mailman/listinfo/glasgow-haskell-users