Allow Unicode escapes in any server encoding, not only UTF-8. SQL includes provisions for numeric Unicode escapes in string literals and identifiers. Previously we only accepted those if they represented ASCII characters or the server encoding was UTF-8, making the conversion to internal form trivial. This patch adjusts things so that we'll call the appropriate encoding conversion function in less-trivial cases, allowing the escape sequence to be accepted so long as it corresponds to some character available in the server encoding.
This also applies to processing of Unicode escapes in JSONB. However, the old restriction still applies to client-side JSON processing, since that hasn't got access to the server's encoding conversion infrastructure. This patch includes some lexer infrastructure that simplifies throwing errors with error cursors pointing into the middle of a string (or other complex token). For the moment I only used it for errors relating to Unicode escapes, but we might later expand the usage to some other cases. Patch by me, reviewed by John Naylor. Discussion: https://postgr.es/m/2393.1578958...@sss.pgh.pa.us Branch ------ master Details ------- https://git.postgresql.org/pg/commitdiff/a6525588b7c21fd8539e9a43ec9c5c245ed1cc91 Modified Files -------------- doc/src/sgml/json.sgml | 19 ++-- doc/src/sgml/syntax.sgml | 100 ++++++++--------- src/backend/parser/parser.c | 74 ++++++------ src/backend/parser/scan.l | 131 +++++++++++++++++++--- src/backend/utils/adt/jsonpath_scan.l | 45 ++------ src/backend/utils/adt/xml.c | 24 +--- src/backend/utils/mb/mbutils.c | 105 +++++++++++++++++ src/common/jsonapi.c | 58 ++++++---- src/include/mb/pg_wchar.h | 17 +++ src/include/parser/scanner.h | 16 +++ src/test/regress/expected/json_encoding.out | 15 +++ src/test/regress/expected/json_encoding_1.out | 39 ++++--- src/test/regress/expected/json_encoding_2.out | 9 ++ src/test/regress/expected/jsonpath_encoding.out | 15 +++ src/test/regress/expected/jsonpath_encoding_1.out | 33 ++++-- src/test/regress/expected/jsonpath_encoding_2.out | 9 ++ src/test/regress/expected/strings.out | 83 +++++++++++++- src/test/regress/sql/json_encoding.sql | 13 ++- src/test/regress/sql/jsonpath_encoding.sql | 13 ++- src/test/regress/sql/strings.sql | 20 ++++ 20 files changed, 612 insertions(+), 226 deletions(-)
