Hi all, On Sun, 4 Dec 2005, Tom Lane wrote:
> Paul Lindner <[EMAIL PROTECTED]> writes: > > To convert your pre-8.1 database to 8.1 you may have to remove and/or > > fix the offending characters. One simple way to fix the problem is to > > run your pg_dump output through the iconv command like this: > > > iconv -c -f UTF8 -t UTF8 -o fixed.sql dump.sql > > Is that really a one-size-fits-all solution? Especially with -c? > It's definately not a one size fits all. The reassuring thing is that others have tried to deal with this problem before. Omar Kilani and I have spent a few hours looking at the problem. For situations where there is a lot of invalid encoding, manual fixing is just not viable. The vim project has a kind of fuzzy encoding conversion which accounts for a lot of the non-UTF8 sequences in UTF8 data. You can use vim to modify your text dump as follows: vim -c ":wq! ++enc=utf8 fixed.dump" original.dump Now, our testing of this is far from exhaustive but it's a lot better than just cutting the data from the original dump. Those suffering the problem should definately check this out, particularly if you have a non-trivial amount of data. Thanks, Gavin ---------------------------(end of broadcast)--------------------------- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match