Hi, On Sat, 14 Feb 2026 at 02:09, Nathan Bossart <[email protected]> wrote: > > Some other random thoughts: > > + match = vector8_or(vector8_eq(chunk, nl), vector8_eq(chunk, cr)); > > + match = vector8_or(vector8_eq(chunk, nl), vector8_eq(chunk, cr)); > > Since \n and \r are well below "normal" ASCII values, I wonder if we could > simplify these to something like > > match = vector8_gt(... vector with all lanes set to \r + 1 ..., chunk);
This didn't work because we have horizontal tab characters whose value (9) is lower than '\r' (13). > + /* Check if we found any special characters */ > + mask = vector8_highbit_mask(match); > + if (mask != 0) > > vector8_highbit_mask() is somewhat expensive on AArch64, so I wonder if > waiting until we enter the "if" block to calculate it has any benefit. I think this makes sense, done. > + simd_hit_eol = (c1 == '\r' || c1 == '\n') && (!is_csv || !in_quote); > > If (is_csv && in_quote), we shouldn't have picked up \r or \n in the first > place, right? You are right, I put an assertion for this. > + simd_hit_eof = c1 == '\\' && c2 == '.' && !is_csv; > + > + /* > + * Do not disable SIMD when we hit EOL or EOF characters. In > + * practice, it does not matter for EOF because parsing ends > + * there, but we keep the behavior consistent. > + */ > + if (!(simd_hit_eof || simd_hit_eol)) > > I'd think that doing less unnecessary work would outweigh the benefits of > consistency for the EOF case. This will work once for the data since SIMD will be disabled afterwards. So, I think this shouldn't affect the performance but I am okay to change if you prefer. I have bencharmed v9 and didn't see any regression. -- Regards, Nazir Bilal Yavuz Microsoft
From 05da2c50de603bd4c2103e8a0c8a37af0599c1f9 Mon Sep 17 00:00:00 2001 From: Nathan Bossart <[email protected]> Date: Wed, 18 Feb 2026 14:55:39 -0600 Subject: [PATCH v9 1/2] Speedup COPY FROM with additional function inlining. Following the example set by commit 58a359e585, we can squeeze out a little more performance from COPY FROM (FORMAT {text,csv}) by forcing CopyReadLineText() to be inlined and by passing the is_csv parameter as a constant. This allows the compiler to emit specialized code with the known-const false comparisons and subsequent branches removed. Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Ayoub Kazar <[email protected]> Discussion: https://postgr.es/m/CAOzEurSW8cNr6TPKsjrstnPfhf4QyQqB4tnPXGGe8N4e_v7Jig%40mail.gmail.com --- src/backend/commands/copyfromparse.c | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-) diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index 94d6f415a06..0aa549837b5 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -141,7 +141,8 @@ static const char BinarySignature[11] = "PGCOPY\n\377\r\n\0"; /* non-export function prototypes */ static bool CopyReadLine(CopyFromState cstate, bool is_csv); -static bool CopyReadLineText(CopyFromState cstate, bool is_csv); +static pg_attribute_always_inline bool CopyReadLineText(CopyFromState cstate, + bool is_csv); static int CopyReadAttributesText(CopyFromState cstate); static int CopyReadAttributesCSV(CopyFromState cstate); static Datum CopyReadBinaryAttribute(CopyFromState cstate, FmgrInfo *flinfo, @@ -1173,8 +1174,18 @@ CopyReadLine(CopyFromState cstate, bool is_csv) resetStringInfo(&cstate->line_buf); cstate->line_buf_valid = false; - /* Parse data and transfer into line_buf */ - result = CopyReadLineText(cstate, is_csv); + /* + * Parse data and transfer into line_buf. + * + * Because this is performance critical, we inline CopyReadLineText() and + * pass the boolean parameters as constants to allow the compiler to emit + * specialized code with the known-const false comparisons and subsequent + * branches removed. + */ + if (is_csv) + result = CopyReadLineText(cstate, true); + else + result = CopyReadLineText(cstate, false); if (result) { @@ -1241,7 +1252,7 @@ CopyReadLine(CopyFromState cstate, bool is_csv) /* * CopyReadLineText - inner loop of CopyReadLine for text mode */ -static bool +static pg_attribute_always_inline bool CopyReadLineText(CopyFromState cstate, bool is_csv) { char *copy_input_buf; -- 2.47.3
From 62d350341ab933d6c0bc3d6a72bc3d1aed36c205 Mon Sep 17 00:00:00 2001 From: Nazir Bilal Yavuz <[email protected]> Date: Fri, 13 Feb 2026 13:28:55 +0300 Subject: [PATCH v9 2/2] Speed up COPY FROM text/CSV parsing using SIMD This patch disables SIMD when SIMD encounters a special character which is neither EOF nor EOL. Author: Shinya Kato <[email protected]> Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Kazar Ayoub <[email protected]> Reviewed-by: Nathan Bossart <[email protected]> Reviewed-by: Neil Conway <[email protected]> Reviewed-by: Andrew Dunstan <[email protected]> Reviewed-by: Manni Wood <[email protected]> Reviewed-by: Mark Wong <[email protected]> Discussion: https://postgr.es/m/CAOzEurSW8cNr6TPKsjrstnPfhf4QyQqB4tnPXGGe8N4e_v7Jig%40mail.gmail.com --- src/backend/commands/copyfrom.c | 3 + src/backend/commands/copyfromparse.c | 135 ++++++++++++++++++++++- src/include/commands/copyfrom_internal.h | 3 + 3 files changed, 137 insertions(+), 4 deletions(-) diff --git a/src/backend/commands/copyfrom.c b/src/backend/commands/copyfrom.c index 2b7556b287c..3dd159f15b2 100644 --- a/src/backend/commands/copyfrom.c +++ b/src/backend/commands/copyfrom.c @@ -1717,6 +1717,9 @@ BeginCopyFrom(ParseState *pstate, cstate->cur_attval = NULL; cstate->relname_only = false; + /* Initialize SIMD */ + cstate->simd_enabled = true; + /* * Allocate buffers for the input pipeline. * diff --git a/src/backend/commands/copyfromparse.c b/src/backend/commands/copyfromparse.c index 0aa549837b5..7ca261a6def 100644 --- a/src/backend/commands/copyfromparse.c +++ b/src/backend/commands/copyfromparse.c @@ -72,6 +72,7 @@ #include "miscadmin.h" #include "pgstat.h" #include "port/pg_bswap.h" +#include "port/simd.h" #include "utils/builtins.h" #include "utils/rel.h" @@ -142,7 +143,8 @@ static const char BinarySignature[11] = "PGCOPY\n\377\r\n\0"; /* non-export function prototypes */ static bool CopyReadLine(CopyFromState cstate, bool is_csv); static pg_attribute_always_inline bool CopyReadLineText(CopyFromState cstate, - bool is_csv); + bool is_csv, + bool simd_enabled); static int CopyReadAttributesText(CopyFromState cstate); static int CopyReadAttributesCSV(CopyFromState cstate); static Datum CopyReadBinaryAttribute(CopyFromState cstate, FmgrInfo *flinfo, @@ -1183,9 +1185,19 @@ CopyReadLine(CopyFromState cstate, bool is_csv) * branches removed. */ if (is_csv) - result = CopyReadLineText(cstate, true); + { + if (cstate->simd_enabled) + result = CopyReadLineText(cstate, true, true); + else + result = CopyReadLineText(cstate, true, false); + } else - result = CopyReadLineText(cstate, false); + { + if (cstate->simd_enabled) + result = CopyReadLineText(cstate, false, true); + else + result = CopyReadLineText(cstate, false, false); + } if (result) { @@ -1253,7 +1265,7 @@ CopyReadLine(CopyFromState cstate, bool is_csv) * CopyReadLineText - inner loop of CopyReadLine for text mode */ static pg_attribute_always_inline bool -CopyReadLineText(CopyFromState cstate, bool is_csv) +CopyReadLineText(CopyFromState cstate, bool is_csv, bool simd_enabled) { char *copy_input_buf; int input_buf_ptr; @@ -1268,6 +1280,14 @@ CopyReadLineText(CopyFromState cstate, bool is_csv) char quotec = '\0'; char escapec = '\0'; +#ifndef USE_NO_SIMD + Vector8 nl = vector8_broadcast('\n'); + Vector8 cr = vector8_broadcast('\r'); + Vector8 bs = vector8_broadcast('\\'); + Vector8 quote = vector8_broadcast(0); + Vector8 escape = vector8_broadcast(0); +#endif + if (is_csv) { quotec = cstate->opts.quote[0]; @@ -1275,6 +1295,12 @@ CopyReadLineText(CopyFromState cstate, bool is_csv) /* ignore special escape processing if it's the same as quotec */ if (quotec == escapec) escapec = '\0'; + +#ifndef USE_NO_SIMD + quote = vector8_broadcast(quotec); + if (quotec != escapec) + escape = vector8_broadcast(escapec); +#endif } /* @@ -1341,6 +1367,107 @@ CopyReadLineText(CopyFromState cstate, bool is_csv) need_data = false; } +#ifndef USE_NO_SIMD + + /* + * Use SIMD instructions to efficiently scan the input buffer for + * special characters (e.g., newline, carriage return, quote, and + * escape). This is faster than byte-by-byte iteration, especially on + * large buffers. + * + * We do not apply the SIMD fast path in either of the following + * cases: - When the previously processed character was an escape + * character (last_was_esc), since the next byte must be examined + * sequentially. - When the remaining buffer is smaller than one + * vector width (sizeof(Vector8)), since SIMD operates on fixed-size + * chunks. + * + * Note that, SIMD may become slower when the input contains many + * special characters. To avoid this regression, we disable SIMD for + * the rest of the input once we encounter a special character which + * is neither EOF nor EOL. + */ + if (simd_enabled && !last_was_esc && copy_buf_len - input_buf_ptr > sizeof(Vector8)) + { + Vector8 chunk; + Vector8 match = vector8_broadcast(0); + + /* Load a chunk of data into a vector register */ + vector8_load(&chunk, (const uint8 *) ©_input_buf[input_buf_ptr]); + + if (is_csv) + { + /* \n and \r are not special inside quotes */ + if (!in_quote) + match = vector8_or(vector8_eq(chunk, nl), vector8_eq(chunk, cr)); + + match = vector8_or(match, vector8_eq(chunk, quote)); + if (escapec != '\0') + match = vector8_or(match, vector8_eq(chunk, escape)); + } + else + { + match = vector8_or(vector8_eq(chunk, nl), vector8_eq(chunk, cr)); + match = vector8_or(match, vector8_eq(chunk, bs)); + } + + /* Check if we found any special characters */ + if (vector8_is_highbit_set(match)) + { + /* + * Found a special character. Advance up to that point and let + * the scalar code handle it. + */ + uint32 mask; + int advance; + char c1, + c2; + bool simd_hit_eol, + simd_hit_eof; + + mask = vector8_highbit_mask(match); + advance = pg_rightmost_one_pos32(mask); + + input_buf_ptr += advance; + c1 = copy_input_buf[input_buf_ptr]; + + /* + * Since we stopped within the chunk and ((copy_buf_len - + * input_buf_ptr) > sizeof(Vector8)) is true, + * copy_input_buf[input_buf_ptr + 1] is guaranteed to be + * readable. + */ + c2 = copy_input_buf[input_buf_ptr + 1]; + + simd_hit_eof = (c1 == '\\' && c2 == '.' && !is_csv); + simd_hit_eol = (c1 == '\r' || c1 == '\n'); + + /* + * If (is_csv && in_quote), we shouldn't have picked up '\r' + * or '\n' in the first place. + */ + Assert(!simd_hit_eol || !(is_csv && in_quote)); + + /* + * Do not disable SIMD when we hit EOL or EOF characters. In + * practice, it does not matter for EOF because parsing ends + * there, but we keep the behavior consistent. + */ + if (!(simd_hit_eof || simd_hit_eol)) + { + simd_enabled = false; + cstate->simd_enabled = false; + } + } + else + { + /* No special characters found, so skip the entire chunk */ + input_buf_ptr += sizeof(Vector8); + continue; + } + } +#endif + /* OK to fetch a character */ prev_raw_ptr = input_buf_ptr; c = copy_input_buf[input_buf_ptr++]; diff --git a/src/include/commands/copyfrom_internal.h b/src/include/commands/copyfrom_internal.h index 822ef33cf69..73ce777c52b 100644 --- a/src/include/commands/copyfrom_internal.h +++ b/src/include/commands/copyfrom_internal.h @@ -89,6 +89,9 @@ typedef struct CopyFromStateData const char *cur_attval; /* current att value for error messages */ bool relname_only; /* don't output line number, att, etc. */ + /* SIMD variables */ + bool simd_enabled; + /* * Working state */ -- 2.47.3
