On Sat, 2006-01-07 at 12:12 +0100, Wiktor Wandachowicz wrote: > Jay Goldman wrote: > > Why do you think the file is correct? > > > > It sounds like 'vi' (aka vim?) did not write a valid utf-8 file ... Maybe > > it was working in some 8bit character set (i.e., a non unicode 256 > > character code set, e.g., ASCII) and used some value in the upper 128 > > characters (the non-unicode part of ASCII) - which would be interpreted by > > unicode processing as a multi-byte character, but the subsequent bytes > > would not be correct. > > > > Cat'ing to the terminal probably won't work as most terminal emulators are > > running with some 8bit character code (e.g., ASCII) with 256 characters and > > not UNICODE. > > Even using 'vi' you probably have to tell it that the file is utf-8 and not > > ASCII. > Maybe you could use the 'iconv' program to make sure that your file > uses a valid utf-8 encoding? Example: >
The file as indeed corrupt in some way. The iconv application said: giconv: illegal input sequence at position 584 I copied the contents of the file with mouse copy paste from one vim to another vim and that solved the problem. Thanks everyone for pointing in the direction of the file format being incorrect. Now all is working ok. > * create the file 'test-vi.txt' with the utf-8 characters > > * create the file 'test-local.txt' in another editor, and > save it in, say, 'ISO-8859-1' encoding > > * run "iconv -f iso-8859-1 -t utf-8 -o test-iconv.txt test-local.txt" > (this should create a file with utf-8 encoding) > > * run "cmp -c test-vi.txt test-iconv.txt" or > "diff -u test-vi.txt test-iconv.txt" > (this should give you warnings if any charcters are different) > > And better yet, you can use the Gnome's 'gedit' - it handles correctly > LOTS of encodings, including Unicode, ISO, DOS- and MS-Win ones. :-) > > Friendly, > Wiktor Wandachowicz > > ------------------------------------------------- > If you wish to unsubscribe from this list, please > send an empty message to the following address: > > [EMAIL PROTECTED] > ------------------------------------------------- > ------------------------------------------------- If you wish to unsubscribe from this list, please send an empty message to the following address: [EMAIL PROTECTED] -------------------------------------------------
