On Mon, Feb 1, 2021 at 12:15 PM Heikki Linnakangas <hlinn...@iki.fi> wrote: > Thanks. I fixed it slightly differently, and also changed LocalToUtf() > to follow the same pattern, even though LocalToUtf() did not have the > same bug.
Looks good to me. > I added a bunch of tests for various built-in conversions. Nice! I would like to have utf8 tests for every category of invalid byte (overlong, surrogate, 5 bytes, etc), but it's not necessary for this patch. > I spent some time refactoring and adding comments all around the patch, > hopefully making it all more clear. One notable difference is that I > renamed 'raw_buf' (which exists in master too) to 'input_buf', and > renamed 'conversion_buf' to 'raw_buf'. I'm going to read through this > patch again another day with fresh eyes, and also try to add some tests > for the corner cases at buffer boundaries. The comments and renaming are really helpful in understanding that file! Although a new patch is likely forthcoming, I did take a brief look and found the following: In copyfromparse.c, this is now out of date: * Read the next input line and stash it in line_buf, with conversion to * server encoding. One of your FIXME comments seems to allude to this, but if we really need a difference here, maybe it should be explained: +#define INPUT_BUF_SIZE 65536 /* we palloc INPUT_BUF_SIZE+1 bytes */ +#define RAW_BUF_SIZE 65536 /* allocated size of the buffer */ Lastly, it looks like pg_do_encoding_conversion_buf() ended up in 0003 accidentally? -- John Naylor EDB: http://www.enterprisedb.com
