I want to index text in UTF-8 format. I use latin characters.
Here are some examples of characters (encoded in ISO-8859-1): ó, é, á, etc.
I used iconv function iconv('ISO-8859-1', 'ASCII//TRANSLIT', 'Animación') and
i got Animaci'on which also contains some break
characters for the tokenizer. Also, for characters like é, á I got 'a, 'e.
The solution is to replace `'` character with some alpha-digit pattern. But
what If I get other break
characters for other latin characters? Or maybe I will use other UTF-8
characters from german language
which also produce some distinct break characters (not alpha-digit characters).
I saw that some people used htmlentities which produce only 2 break characters
('&' and ';'). In this case I
can find 2 alphadigit patterns to match them more easily. And htmlentities
encode all utf-8 characters. Is this
the best solution? Maybe there are some Analyzers which I can use and which not
break on '&' and ';' characters.
Maybe someone has a better solution or some opinions on this problem.
Thank you.
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
