Dear Karl, It finally worked and I would like to thank you for your patience and your help. I could apply it to any text read from the standard input and avoid the CO options with these three lines in the beginning of the program.
use utf8; binmode(STDOUT, ":encoding(UTF-8)"); binmode(STDIN, ":encoding(UTF-8)"); The program must read the standard input explicitly, which means that perl token_perl.pl text.txt will not work. It has to be perl token_perl.pl < text.txt instead. Also, it is also pretty curious that the program worked fine in its original form, for all the characters with the exception of the quote » and that the problem disappeared when I removed the û character from the tr// list. Now everything works fine without this hack. Thank you!!!! Pierre
