> Chars 145 - 150 (the fancy quotes, the middot and the long minus MS Word
> produces), are
> causing XSL transformation problems.
> 
> Basically what I try to do, is to convert a newsitem into an email, for
> 'email this article'. This
> needs output: text and preferably ASCII or ISO-8859-1 output encoding.
> 
> Every time an article contains one of these chars, Sablotron chokes on an
> invalid token and
> if I try to 'iconv' these files, the same thing happens.
> 
> Is there any other way to do this maybe - I've tried both ISO-8859-1 as
> windows-1252 as
> input encoding, but neither work.

Characters 127-159 are supposed to be reserved for control characters
according to ISO-8859-1. Unfortunately, Microsoft doesn't believe in
following standards not created by Microsoft, so they filled up this
character space with actual characters in Windows. Little did they realize
how much of a problem this would create once the internet was born and their
operating system was forced to interact with standards-compliant operating
systems like UNIX.

Anyway, my favorite kludge to this problem (although it might not be the
most elegant) is to simply convert these bastard characters into html
entities or normal characters:

127 = x7F -> control character (ignore)
128 = x80 -> control character (ignore)
129 = x81 -> control character (ignore)
130 = x82 -> ' (apostrophe used as single quote)
131 = x83 -> <i>f</i>
132 = x84 -> &quot;
133 = x85 -> ...
134 = x86 -> &sup1;
135 = x87 -> &sup2;
136 = x88 -> ^
137 = x89 -> o/oo
138 = x8A -> Sh
139 = x8B -> <
140 = x8C -> Oe
141 = x8D -> control character (ignore)
142 = x8E -> control character (ignore)
143 = x8F -> control character (ignore)
144 = x90 -> control character (ignore)
145 = x91 -> ' (apostrophe used as single quote)
146 = x92 -> ' (apostrophe used as single quote)
147 = x93 -> &quot;
148 = x94 -> &quot;
149 = x95 -> *
150 = x96 -> -
151 = x97 -> --
152 = x98 -> ~
153 = x99 -> <sup>(TM)</sup>
154 = x9A -> sh
155 = x9B -> >
156 = x9C -> oe
157 = x9D -> control character (ignore)
158 = x9E -> control character (ignore)
159 = x9F -> Y

Characters 160-255 are safe, but should probably be converted into HTML
entities to be safer. These can be converted directly:

160 = xA0 -> &#160; or &#xA0;
161 = xA1 -> &#161; or &#xA1;
etc...

Anything above 255 (or lower than 32) should probably be ignored.

If anyone has additional (or better) suggestions for this problem please let
me know as I'm very interested in this issue.

Ryan Smith
[EMAIL PROTECTED]

Reply via email to