Hi, Here is a rebase of this experimental patch. I think the basic mechanics are promising, but we haven't agreed on a UX. I hope we can figure this out.
Restating the choice made in this branch of the experiment: Here I try to be just like DB2 (if I understood its manual correctly). In DB2, you can use names like "en_US" if you don't care about changes, and names like "CLDR181_en_US" if you do. It's the user's choice to use the second kind to avoid "unexpected effects on applications or database objects" after upgrades. Translated to PostgreSQL concepts, you can use a database default ICU locale like "en-US" if you don't care and "67:en-US" if you do, and for COLLATION objects it's the same. The convention I tried in this patch is that you use either "en-US-x-icu" (which points to "en-US") or "en-US-x-icu67" (which points to "67:en-US") depending on whether you care about this problem. I recognise that this is a bit cheesy, it's all the user's problem to deal with or ignore. An alternative mentioned by Peter E was that the locale names shouldn't carry the prefix, but somehow we should have a list of ICU versions to search for a matching datcollversion/collversion. How would that look? Perhaps a GUC, icu_library_versions = '63, 67, 71'? There is a currently natural and smallish range of supported versions, probably something like 54 ... U_ICU_VERSION_MAJOR_NUM, but it seems a bit weird to try to dlopen ~25 libraries or whatever it might be... Do you think we should try to code this up? I haven't tried it, but the main usability problem I predict with that idea is this: It can cope with a scenario where you created a database with ICU 63 and started using a default of "en" and maybe some explicit fr-x-icu or whatever, and then you upgrade to a new postgres binary using ICU 71, and, as long as you still have ICU 63 installed it'll just magicaly keep using 63, now via dlopen(). But it doesn't provide a way for me to create a new database that uses 63 on purpose when I know what I'm doing. There are various reasons I might want to do that. Maybe the ideas could be combined? Perhaps "en" means "create using binary's linked ICU, open using search-by-collversion", while "67:en" explicitly says which to use? Changes since last version: * Now it just uses the default dlopen() search path, unless you set icu_library_path. Is that a security problem? It's pretty convenient, because it means you can just "apt-get install libicu63" (or local equivalent) and that's all, now 63 is available. * To try the idea out, I made it automatically create "*-x-icu67" alongside the regular "-x-icu" collation objects at initdb time.
From d3e83d0aa5cbb3eb192a2f66d68623cd3b1595b4 Mon Sep 17 00:00:00 2001 From: Thomas Munro <thomas.munro@gmail.com> Date: Wed, 8 Jun 2022 17:43:53 +1200 Subject: [PATCH v5] WIP: Multi-version ICU. Add a layer of indirection when accessing ICU, so that multiple major versions of the library can be used at once. Versions other than the one that PostgreSQL was linked against are opened with dlopen(), but we refuse to open version higher than the one were were compiled against. The ABI might change in future releases so that wouldn't be safe. By default, the system linker's default search path is used to find libraries, but icu_library_path may be used to specify an absolute path to look in. ICU libraries are expected to have been built without ICU's --disable-renaming option. That is, major versions must use distinct symbol names. This arrangement means that at least one major version of ICU is always available -- the one that PostgreSQL was linked again. It should be simple on most software distributions to install extra versions using a package manager, or to build extra libraries as required, to access older ICU releases. For example, on Debian bullseye the packages are named libicu63, libicu67, libicu71. In this version of the patch, '63:en' used as a database default locale or COLLATION object requests ICU library 63, and 'en' requests the library that is linked against the postgres executable. XXX Many other designs possible, to discuss! Discussion: https://postgr.es/m/CA%2BhUKGL4VZRpP3CkjYQkv4RQ6pRYkPkSNgKSxFBwciECQ0mEuQ%40mail.gmail.com --- src/backend/access/hash/hashfunc.c | 16 +- src/backend/commands/collationcmds.c | 20 ++ src/backend/utils/adt/formatting.c | 53 +++- src/backend/utils/adt/pg_locale.c | 364 ++++++++++++++++++++++++++- src/backend/utils/adt/varchar.c | 16 +- src/backend/utils/adt/varlena.c | 56 +++-- src/backend/utils/misc/guc_tables.c | 14 ++ src/include/utils/pg_locale.h | 73 ++++++ src/tools/pgindent/typedefs.list | 3 + 9 files changed, 549 insertions(+), 66 deletions(-) diff --git a/src/backend/access/hash/hashfunc.c b/src/backend/access/hash/hashfunc.c index b57ed946c4..0a61538efd 100644 --- a/src/backend/access/hash/hashfunc.c +++ b/src/backend/access/hash/hashfunc.c @@ -298,11 +298,11 @@ hashtext(PG_FUNCTION_ARGS) ulen = icu_to_uchar(&uchar, VARDATA_ANY(key), VARSIZE_ANY_EXHDR(key)); - bsize = ucol_getSortKey(mylocale->info.icu.ucol, - uchar, ulen, NULL, 0); + bsize = PG_ICU_LIB(mylocale)->getSortKey(PG_ICU_COL(mylocale), + uchar, ulen, NULL, 0); buf = palloc(bsize); - ucol_getSortKey(mylocale->info.icu.ucol, - uchar, ulen, buf, bsize); + PG_ICU_LIB(mylocale)->getSortKey(PG_ICU_COL(mylocale), + uchar, ulen, buf, bsize); result = hash_any(buf, bsize); @@ -355,11 +355,11 @@ hashtextextended(PG_FUNCTION_ARGS) ulen = icu_to_uchar(&uchar, VARDATA_ANY(key), VARSIZE_ANY_EXHDR(key)); - bsize = ucol_getSortKey(mylocale->info.icu.ucol, - uchar, ulen, NULL, 0); + bsize = PG_ICU_LIB(mylocale)->getSortKey(PG_ICU_COL(mylocale), + uchar, ulen, NULL, 0); buf = palloc(bsize); - ucol_getSortKey(mylocale->info.icu.ucol, - uchar, ulen, buf, bsize); + PG_ICU_LIB(mylocale)->getSortKey(PG_ICU_COL(mylocale), + uchar, ulen, buf, bsize); result = hash_any_extended(buf, bsize, PG_GETARG_INT64(1)); diff --git a/src/backend/commands/collationcmds.c b/src/backend/commands/collationcmds.c index fcfc02d2ae..26e747d9d7 100644 --- a/src/backend/commands/collationcmds.c +++ b/src/backend/commands/collationcmds.c @@ -812,6 +812,26 @@ pg_import_system_collations(PG_FUNCTION_ARGS) CreateComments(collid, CollationRelationId, 0, icucomment); } + + /* Also create an object pinned to an ICU major version. */ + collid = CollationCreate(psprintf("%s-x-icu-%d", langtag, U_ICU_VERSION_MAJOR_NUM), + nspid, GetUserId(), + COLLPROVIDER_ICU, true, -1, + NULL, NULL, + psprintf("%d:%s", U_ICU_VERSION_MAJOR_NUM, iculocstr), + get_collation_actual_version(COLLPROVIDER_ICU, iculocstr), + true, true); + if (OidIsValid(collid)) + { + ncreated++; + + CommandCounterIncrement(); + + icucomment = get_icu_locale_comment(name); + if (icucomment) + CreateComments(collid, CollationRelationId, 0, + icucomment); + } } } #endif /* USE_ICU */ diff --git a/src/backend/utils/adt/formatting.c b/src/backend/utils/adt/formatting.c index 26f498b5df..0c3c7724d7 100644 --- a/src/backend/utils/adt/formatting.c +++ b/src/backend/utils/adt/formatting.c @@ -1599,6 +1599,11 @@ typedef int32_t (*ICU_Convert_Func) (UChar *dest, int32_t destCapacity, const UChar *src, int32_t srcLength, const char *locale, UErrorCode *pErrorCode); +typedef int32_t (*ICU_Convert_BI_Func) (UChar *dest, int32_t destCapacity, + const UChar *src, int32_t srcLength, + UBreakIterator *bi, + const char *locale, + UErrorCode *pErrorCode); static int32_t icu_convert_case(ICU_Convert_Func func, pg_locale_t mylocale, @@ -1623,18 +1628,41 @@ icu_convert_case(ICU_Convert_Func func, pg_locale_t mylocale, } if (U_FAILURE(status)) ereport(ERROR, - (errmsg("case conversion failed: %s", u_errorName(status)))); + (errmsg("case conversion failed: %s", + PG_ICU_LIB(mylocale)->errorName(status)))); return len_dest; } +/* + * Like icu_convert_case, but func takes a break iterator (which we don't + * make use of). + */ static int32_t -u_strToTitle_default_BI(UChar *dest, int32_t destCapacity, - const UChar *src, int32_t srcLength, - const char *locale, - UErrorCode *pErrorCode) +icu_convert_case_bi(ICU_Convert_BI_Func func, pg_locale_t mylocale, + UChar **buff_dest, UChar *buff_source, int32_t len_source) { - return u_strToTitle(dest, destCapacity, src, srcLength, - NULL, locale, pErrorCode); + UErrorCode status; + int32_t len_dest; + + len_dest = len_source; /* try first with same length */ + *buff_dest = palloc(len_dest * sizeof(**buff_dest)); + status = U_ZERO_ERROR; + len_dest = func(*buff_dest, len_dest, buff_source, len_source, NULL, + mylocale->info.icu.locale, &status); + if (status == U_BUFFER_OVERFLOW_ERROR) + { + /* try again with adjusted length */ + pfree(*buff_dest); + *buff_dest = palloc(len_dest * sizeof(**buff_dest)); + status = U_ZERO_ERROR; + len_dest = func(*buff_dest, len_dest, buff_source, len_source, NULL, + mylocale->info.icu.locale, &status); + } + if (U_FAILURE(status)) + ereport(ERROR, + (errmsg("case conversion failed: %s", + PG_ICU_LIB(mylocale)->errorName(status)))); + return len_dest; } #endif /* USE_ICU */ @@ -1702,7 +1730,8 @@ str_tolower(const char *buff, size_t nbytes, Oid collid) UChar *buff_conv; len_uchar = icu_to_uchar(&buff_uchar, buff, nbytes); - len_conv = icu_convert_case(u_strToLower, mylocale, + len_conv = icu_convert_case(PG_ICU_LIB(mylocale)->strToLower, + mylocale, &buff_conv, buff_uchar, len_uchar); icu_from_uchar(&result, buff_conv, len_conv); pfree(buff_uchar); @@ -1824,7 +1853,8 @@ str_toupper(const char *buff, size_t nbytes, Oid collid) UChar *buff_conv; len_uchar = icu_to_uchar(&buff_uchar, buff, nbytes); - len_conv = icu_convert_case(u_strToUpper, mylocale, + len_conv = icu_convert_case(PG_ICU_LIB(mylocale)->strToUpper, + mylocale, &buff_conv, buff_uchar, len_uchar); icu_from_uchar(&result, buff_conv, len_conv); pfree(buff_uchar); @@ -1947,8 +1977,9 @@ str_initcap(const char *buff, size_t nbytes, Oid collid) UChar *buff_conv; len_uchar = icu_to_uchar(&buff_uchar, buff, nbytes); - len_conv = icu_convert_case(u_strToTitle_default_BI, mylocale, - &buff_conv, buff_uchar, len_uchar); + len_conv = icu_convert_case_bi(PG_ICU_LIB(mylocale)->strToTitle, + mylocale, + &buff_conv, buff_uchar, len_uchar); icu_from_uchar(&result, buff_conv, len_conv); pfree(buff_uchar); pfree(buff_conv); diff --git a/src/backend/utils/adt/pg_locale.c b/src/backend/utils/adt/pg_locale.c index 2b42d9ccd8..bf76516406 100644 --- a/src/backend/utils/adt/pg_locale.c +++ b/src/backend/utils/adt/pg_locale.c @@ -58,6 +58,7 @@ #include "catalog/pg_collation.h" #include "catalog/pg_control.h" #include "mb/pg_wchar.h" +#include "miscadmin.h" #include "utils/builtins.h" #include "utils/formatting.h" #include "utils/guc_hooks.h" @@ -69,6 +70,7 @@ #ifdef USE_ICU #include <unicode/ucnv.h> +#include <unicode/ustring.h> #endif #ifdef __GLIBC__ @@ -79,14 +81,31 @@ #include <shlwapi.h> #endif +#include <dlfcn.h> + #define MAX_L10N_DATA 80 +#ifdef USE_ICU + +/* + * We don't want to call into dlopen'd ICU libraries that are newer than the + * one we were compiled and linked against, just in case there is an + * incompatible API change. + */ +#define PG_MAX_ICU_MAJOR_VERSION U_ICU_VERSION_MAJOR_NUM + +/* An old ICU release that we know has the right API. */ +#define PG_MIN_ICU_MAJOR_VERSION 54 + +#endif + /* GUC settings */ char *locale_messages; char *locale_monetary; char *locale_numeric; char *locale_time; +char *icu_library_path; /* * lc_time localization cache. @@ -1398,29 +1417,343 @@ lc_ctype_is_c(Oid collation) return (lookup_collation_cache(collation, true))->ctype_is_c; } +#ifdef USE_ICU + struct pg_locale_struct default_locale; +/* Linked list of ICU libraries we have loaded. */ +static pg_icu_library *icu_library_list = NULL; + +/* + * Free an ICU library. pg_icu_library objects that are successfully + * constructed stick around for the lifetime of the backend, but this is used + * to clean up if initialization fails. + */ +static void +free_icu_library(pg_icu_library *lib) +{ + if (lib->libicui18n_handle) + dlclose(lib->libicui18n_handle); + if (lib->libicuuc_handle) + dlclose(lib->libicuuc_handle); + pfree(lib); +} + +static void * +get_icu_function(void *handle, const char *function, int version) +{ + char name[80]; + + snprintf(name, sizeof(name), "%s_%d", function, version); + + return dlsym(handle, name); +} + +/* + * Probe a dynamically loaded library to see which major version of ICU it + * contains. + */ +static int +get_icu_library_major_version(void *handle) +{ + for (int i = PG_MIN_ICU_MAJOR_VERSION; i <= PG_MAX_ICU_MAJOR_VERSION; ++i) + if (get_icu_function(handle, "ucol_open", i) || + get_icu_function(handle, "u_strToUpper", i)) + return i; + + /* + * It's a later version we don't dare use, an old version we don't + * support, an ICU build with symbol suffixes disabled, or not ICU. + */ + return -1; +} + +/* + * We have to load a couple of different libraries, so we'll reuse the code to + * do that. + */ +static void * +load_icu_library(pg_icu_library *lib, const char *name) +{ + void *handle; + int found_major_version; + + handle = dlopen(name, RTLD_NOW | RTLD_GLOBAL); + if (handle == NULL) + { + int errno_save = errno; + + free_icu_library(lib); + errno = errno_save; + + ereport(ERROR, + (errmsg("could not load library \"%s\": %m", name))); + } + + found_major_version = get_icu_library_major_version(handle); + if (found_major_version < 0) + { + free_icu_library(lib); + ereport(ERROR, + (errmsg("could not find compatible ICU major version in library \"%s\"", + name))); + } + + if (found_major_version != lib->major_version) + { + free_icu_library(lib); + ereport(ERROR, + (errmsg("expected to find ICU major version %d in library \"%s\", but found %d", + lib->major_version, name, found_major_version))); + } + + return handle; +} + +/* + * Given an ICU major version number, return the object we need to access it, + * or fail while trying to load it. + */ +static pg_icu_library * +get_icu_library(int major_version) +{ + pg_icu_library *lib; + + Assert(major_version >= PG_MIN_ICU_MAJOR_VERSION && + major_version <= PG_MAX_ICU_MAJOR_VERSION); + + /* Try to find it in our list of existing libraries. */ + for (lib = icu_library_list; lib; lib = lib->next) + if (lib->major_version == major_version) + return lib; + + /* Make a new entry. */ + lib = MemoryContextAllocZero(TopMemoryContext, sizeof(*lib)); + if (major_version == U_ICU_VERSION_MAJOR_NUM) + { + /* + * This is the version we were compiled and linked against. Simply + * assign the function pointers. + * + * These assignments will fail to compile if an incompatible API + * change is made to some future version of ICU, at which point we + * might need to consider special treatment for different major + * version ranges, with intermediate trampoline functions. + */ + lib->major_version = major_version; + lib->open = ucol_open; + lib->close = ucol_close; + lib->getVersion = ucol_getVersion; + lib->versionToString = u_versionToString; + lib->strcoll = ucol_strcoll; + lib->strcollUTF8 = ucol_strcollUTF8; + lib->getSortKey = ucol_getSortKey; + lib->nextSortKeyPart = ucol_nextSortKeyPart; + lib->setUTF8 = uiter_setUTF8; + lib->errorName = u_errorName; + lib->strToUpper = u_strToUpper; + lib->strToLower = u_strToLower; + lib->strToTitle = u_strToTitle; + + /* + * Also assert the size of a couple of types used as output buffers, + * as a canary to tell us to add extra padding in the (unlikely) event + * that a later release makes these values smaller. + */ + StaticAssertStmt(U_MAX_VERSION_STRING_LENGTH == 20, + "u_versionToString output buffer size changed incompatibly"); + StaticAssertStmt(U_MAX_VERSION_LENGTH == 4, + "ucol_getVersion output buffer size changed incompatibly"); + } + else + { + /* This is an older version, so we'll need to use dlopen(). */ + char libicui18n_name[MAXPGPATH]; + char libicuuc_name[MAXPGPATH]; + + /* + * We don't like to open versions newer than what we're linked + * against, to reduce the risk of an API change biting us. + */ + if (major_version > U_ICU_VERSION_MAJOR_NUM) + elog(ERROR, "ICU major version %d higher than linked version %d, refusing to open", + major_version, U_ICU_VERSION_MAJOR_NUM); + + lib->major_version = major_version; + + /* + * See + * https://unicode-org.github.io/icu/userguide/icu4c/packaging.html#icu-versions + * for conventions on library naming on POSIX and Windows systems. + */ + + /* Load the collation library. */ + snprintf(libicui18n_name, + sizeof(libicui18n_name), +#ifdef WIN32 + "%s%sicui18n%d." DLSUFFIX, + icu_library_path, + icu_library_path[0] ? "\\" : "", +#else + "%s%slibicui18n" DLSUFFIX ".%d", + icu_library_path, + icu_library_path[0] ? "/" : "", +#endif + major_version); + lib->libicui18n_handle = load_icu_library(lib, libicui18n_name); + + /* Load the ctype library. */ + snprintf(libicuuc_name, + sizeof(libicuuc_name), +#ifdef WIN32 + "%s%sicuuc%d." DLSUFFIX, + icu_library_path, + icu_library_path[0] ? "\\" : "", +#else + "%s%slibicuuc" DLSUFFIX ".%d", + icu_library_path, + icu_library_path[0] ? "/" : "", +#endif + major_version); + lib->libicuuc_handle = load_icu_library(lib, libicuuc_name); + + /* Look up all the functions we need. */ + lib->open = get_icu_function(lib->libicui18n_handle, + "ucol_open", + major_version); + lib->close = get_icu_function(lib->libicui18n_handle, + "ucol_close", + major_version); + lib->getVersion = get_icu_function(lib->libicui18n_handle, + "ucol_getVersion", + major_version); + lib->versionToString = get_icu_function(lib->libicui18n_handle, + "u_versionToString", + major_version); + lib->strcoll = get_icu_function(lib->libicui18n_handle, + "ucol_strcoll", + major_version); + lib->strcollUTF8 = get_icu_function(lib->libicui18n_handle, + "ucol_strcollUTF8", + major_version); + lib->getSortKey = get_icu_function(lib->libicui18n_handle, + "ucol_getSortKey", + major_version); + lib->nextSortKeyPart = get_icu_function(lib->libicui18n_handle, + "ucol_nextSortKeyPart", + major_version); + lib->setUTF8 = get_icu_function(lib->libicui18n_handle, + "uiter_setUTF8", + major_version); + lib->errorName = get_icu_function(lib->libicui18n_handle, + "u_errorName", + major_version); + lib->strToUpper = get_icu_function(lib->libicuuc_handle, + "u_strToUpper", + major_version); + lib->strToLower = get_icu_function(lib->libicuuc_handle, + "u_strToLower", + major_version); + lib->strToTitle = get_icu_function(lib->libicuuc_handle, + "u_strToTitle", + major_version); + if (!lib->open || + !lib->close || + !lib->getVersion || + !lib->versionToString || + !lib->strcoll || + !lib->strcollUTF8 || + !lib->getSortKey || + !lib->nextSortKeyPart || + !lib->setUTF8 || + !lib->errorName || + !lib->strToUpper || + !lib->strToLower || + !lib->strToTitle) + { + free_icu_library(lib); + ereport(ERROR, + (errmsg("could not find expected symbols in library \"%s\"", + libicui18n_name))); + } + } + + lib->next = icu_library_list; + icu_library_list = lib; + + return lib; +} + +/* + * Look up the library to use for a given collcollate string. + */ +static pg_icu_library * +get_icu_library_for_collation(const char *collcollate, const char **rest) +{ + int major_version; + char *separator; + char *after_prefix; + + separator = strchr(collcollate, ':'); + + /* + * If it's a traditional value without a prefix, use the library we are + * linked against. + */ + if (separator == NULL) + { + *rest = collcollate; + return get_icu_library(U_ICU_VERSION_MAJOR_NUM); + } + + /* If it has a prefix, interpret it as an ICU major version. */ + major_version = strtol(collcollate, &after_prefix, 10); + if (after_prefix != separator) + elog(ERROR, + "could not parse ICU major library version: \"%s\"", + collcollate); + if (major_version < PG_MIN_ICU_MAJOR_VERSION || + major_version > PG_MAX_ICU_MAJOR_VERSION) + elog(ERROR, + "ICU major library verision out of supported range: \"%s\"", + collcollate); + + /* The part after the separate will be passed to the library. */ + *rest = separator + 1; + + return get_icu_library(major_version); +} + +#endif + void make_icu_collator(const char *iculocstr, struct pg_locale_struct *resultp) { #ifdef USE_ICU + pg_icu_library *lib; UCollator *collator; UErrorCode status; + lib = get_icu_library_for_collation(iculocstr, &iculocstr); status = U_ZERO_ERROR; - collator = ucol_open(iculocstr, &status); + collator = lib->open(iculocstr, &status); if (U_FAILURE(status)) ereport(ERROR, (errmsg("could not open collator for locale \"%s\": %s", - iculocstr, u_errorName(status)))); + iculocstr, lib->errorName(status)))); - if (U_ICU_VERSION_MAJOR_NUM < 54) + /* + * XXX can we just drop this cruft and make 54 the minimum supported + * version? + */ + if (lib->major_version < 54) icu_set_collation_attributes(collator, iculocstr); /* We will leak this string if the caller errors later :-( */ resultp->info.icu.locale = MemoryContextStrdup(TopMemoryContext, iculocstr); resultp->info.icu.ucol = collator; + resultp->info.icu.lib = lib; #else /* not USE_ICU */ /* could get here if a collation was created by a build with ICU */ ereport(ERROR, @@ -1651,21 +1984,23 @@ get_collation_actual_version(char collprovider, const char *collcollate) #ifdef USE_ICU if (collprovider == COLLPROVIDER_ICU) { + pg_icu_library *lib; UCollator *collator; UErrorCode status; UVersionInfo versioninfo; char buf[U_MAX_VERSION_STRING_LENGTH]; + lib = get_icu_library_for_collation(collcollate, &collcollate); status = U_ZERO_ERROR; - collator = ucol_open(collcollate, &status); + collator = lib->open(collcollate, &status); if (U_FAILURE(status)) ereport(ERROR, (errmsg("could not open collator for locale \"%s\": %s", - collcollate, u_errorName(status)))); - ucol_getVersion(collator, versioninfo); - ucol_close(collator); + collcollate, lib->errorName(status)))); + lib->getVersion(collator, versioninfo); + lib->close(collator); - u_versionToString(versioninfo, buf); + lib->versionToString(versioninfo, buf); collversion = pstrdup(buf); } else @@ -1733,6 +2068,8 @@ get_collation_actual_version(char collprovider, const char *collcollate) #ifdef USE_ICU + + /* * Converter object for converting between ICU's UChar strings and C strings * in database encoding. Since the database encoding doesn't change, we only @@ -1954,19 +2291,22 @@ void check_icu_locale(const char *icu_locale) { #ifdef USE_ICU + pg_icu_library *lib; UCollator *collator; UErrorCode status; + lib = get_icu_library_for_collation(icu_locale, &icu_locale); status = U_ZERO_ERROR; - collator = ucol_open(icu_locale, &status); + collator = lib->open(icu_locale, &status); if (U_FAILURE(status)) ereport(ERROR, (errmsg("could not open collator for locale \"%s\": %s", - icu_locale, u_errorName(status)))); + icu_locale, lib->errorName(status)))); - if (U_ICU_VERSION_MAJOR_NUM < 54) + /* XXX can we just drop this cruft? */ + if (lib->major_version < 54) icu_set_collation_attributes(collator, icu_locale); - ucol_close(collator); + lib->close(collator); #else ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), diff --git a/src/backend/utils/adt/varchar.c b/src/backend/utils/adt/varchar.c index 68e2e6f7a7..e0c86870e0 100644 --- a/src/backend/utils/adt/varchar.c +++ b/src/backend/utils/adt/varchar.c @@ -1026,11 +1026,11 @@ hashbpchar(PG_FUNCTION_ARGS) ulen = icu_to_uchar(&uchar, keydata, keylen); - bsize = ucol_getSortKey(mylocale->info.icu.ucol, - uchar, ulen, NULL, 0); + bsize = PG_ICU_LIB(mylocale)->getSortKey(PG_ICU_COL(mylocale), + uchar, ulen, NULL, 0); buf = palloc(bsize); - ucol_getSortKey(mylocale->info.icu.ucol, - uchar, ulen, buf, bsize); + PG_ICU_LIB(mylocale)->getSortKey(PG_ICU_COL(mylocale), + uchar, ulen, buf, bsize); result = hash_any(buf, bsize); @@ -1087,11 +1087,11 @@ hashbpcharextended(PG_FUNCTION_ARGS) ulen = icu_to_uchar(&uchar, VARDATA_ANY(key), VARSIZE_ANY_EXHDR(key)); - bsize = ucol_getSortKey(mylocale->info.icu.ucol, - uchar, ulen, NULL, 0); + bsize = PG_ICU_LIB(mylocale)->getSortKey(PG_ICU_COL(mylocale), + uchar, ulen, NULL, 0); buf = palloc(bsize); - ucol_getSortKey(mylocale->info.icu.ucol, - uchar, ulen, buf, bsize); + PG_ICU_LIB(mylocale)->getSortKey(PG_ICU_COL(mylocale), + uchar, ulen, buf, bsize); result = hash_any_extended(buf, bsize, PG_GETARG_INT64(1)); diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index c5e7ee7ca2..cf891a5654 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -1667,13 +1667,14 @@ varstr_cmp(const char *arg1, int len1, const char *arg2, int len2, Oid collid) UErrorCode status; status = U_ZERO_ERROR; - result = ucol_strcollUTF8(mylocale->info.icu.ucol, - arg1, len1, - arg2, len2, - &status); + result = PG_ICU_LIB(mylocale)->strcollUTF8(PG_ICU_COL(mylocale), + arg1, len1, + arg2, len2, + &status); if (U_FAILURE(status)) ereport(ERROR, - (errmsg("collation failed: %s", u_errorName(status)))); + (errmsg("collation failed: %s", + PG_ICU_LIB(mylocale)->errorName(status)))); } else #endif @@ -1686,9 +1687,9 @@ varstr_cmp(const char *arg1, int len1, const char *arg2, int len2, Oid collid) ulen1 = icu_to_uchar(&uchar1, arg1, len1); ulen2 = icu_to_uchar(&uchar2, arg2, len2); - result = ucol_strcoll(mylocale->info.icu.ucol, - uchar1, ulen1, - uchar2, ulen2); + result = PG_ICU_LIB(mylocale)->strcoll(PG_ICU_COL(mylocale), + uchar1, ulen1, + uchar2, ulen2); pfree(uchar1); pfree(uchar2); @@ -2388,13 +2389,14 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup) UErrorCode status; status = U_ZERO_ERROR; - result = ucol_strcollUTF8(sss->locale->info.icu.ucol, - a1p, len1, - a2p, len2, - &status); + result = PG_ICU_LIB(sss->locale)->strcollUTF8(PG_ICU_COL(sss->locale), + a1p, len1, + a2p, len2, + &status); if (U_FAILURE(status)) ereport(ERROR, - (errmsg("collation failed: %s", u_errorName(status)))); + (errmsg("collation failed: %s", + PG_ICU_LIB(sss->locale)->errorName(status)))); } else #endif @@ -2407,9 +2409,9 @@ varstrfastcmp_locale(char *a1p, int len1, char *a2p, int len2, SortSupport ssup) ulen1 = icu_to_uchar(&uchar1, a1p, len1); ulen2 = icu_to_uchar(&uchar2, a2p, len2); - result = ucol_strcoll(sss->locale->info.icu.ucol, - uchar1, ulen1, - uchar2, ulen2); + result = PG_ICU_LIB(sss->locale)->strcoll(PG_ICU_COL(sss->locale), + uchar1, ulen1, + uchar2, ulen2); pfree(uchar1); pfree(uchar2); @@ -2569,24 +2571,24 @@ varstr_abbrev_convert(Datum original, SortSupport ssup) uint32_t state[2]; UErrorCode status; - uiter_setUTF8(&iter, sss->buf1, len); + PG_ICU_LIB(sss->locale)->setUTF8(&iter, sss->buf1, len); state[0] = state[1] = 0; /* won't need that again */ status = U_ZERO_ERROR; - bsize = ucol_nextSortKeyPart(sss->locale->info.icu.ucol, - &iter, - state, - (uint8_t *) sss->buf2, - Min(sizeof(Datum), sss->buflen2), - &status); + bsize = PG_ICU_LIB(sss->locale)->nextSortKeyPart(PG_ICU_COL(sss->locale), + &iter, + state, + (uint8_t *) sss->buf2, + Min(sizeof(Datum), sss->buflen2), + &status); if (U_FAILURE(status)) ereport(ERROR, (errmsg("sort key generation failed: %s", - u_errorName(status)))); + PG_ICU_LIB(sss->locale)->errorName(status)))); } else - bsize = ucol_getSortKey(sss->locale->info.icu.ucol, - uchar, ulen, - (uint8_t *) sss->buf2, sss->buflen2); + bsize = PG_ICU_LIB(sss->locale)->getSortKey(PG_ICU_COL(sss->locale), + uchar, ulen, + (uint8_t *) sss->buf2, sss->buflen2); } else #endif diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c index 05ab087934..e60081c384 100644 --- a/src/backend/utils/misc/guc_tables.c +++ b/src/backend/utils/misc/guc_tables.c @@ -3922,6 +3922,20 @@ struct config_string ConfigureNamesString[] = NULL, NULL, NULL }, + { + {"icu_library_path", PGC_SUSET, CLIENT_CONN_OTHER, + gettext_noop("Sets the path for dynamically loadable ICU libraries."), + gettext_noop("If versions of ICU other than the one that " + "PostgreSQL is linked against, they will be open " + "from this path. If empty, the system linker search " + "path will be used."), + GUC_SUPERUSER_ONLY + }, + &icu_library_path, + "", + NULL, NULL, NULL + }, + { {"krb_server_keyfile", PGC_SIGHUP, CONN_AUTH_AUTH, gettext_noop("Sets the location of the Kerberos server key file."), diff --git a/src/include/utils/pg_locale.h b/src/include/utils/pg_locale.h index a875942123..59613e4f56 100644 --- a/src/include/utils/pg_locale.h +++ b/src/include/utils/pg_locale.h @@ -17,6 +17,7 @@ #endif #ifdef USE_ICU #include <unicode/ucol.h> +#include <unicode/ubrk.h> #endif #ifdef USE_ICU @@ -40,6 +41,7 @@ extern PGDLLIMPORT char *locale_messages; extern PGDLLIMPORT char *locale_monetary; extern PGDLLIMPORT char *locale_numeric; extern PGDLLIMPORT char *locale_time; +extern PGDLLIMPORT char *icu_library_path; /* lc_time localization cache */ extern PGDLLIMPORT char *localized_abbrev_days[]; @@ -63,6 +65,71 @@ extern struct lconv *PGLC_localeconv(void); extern void cache_locale_time(void); +#ifdef USE_ICU + +/* + * An ICU library version that we're either linked against or have loaded at + * runtime. + */ +typedef struct pg_icu_library +{ + int major_version; + void *libicui18n_handle; + void *libicuuc_handle; + UCollator *(*open) (const char *loc, UErrorCode *status); + void (*close) (UCollator *coll); + void (*getVersion) (const UCollator *coll, UVersionInfo info); + void (*versionToString) (const UVersionInfo versionArray, + char *versionString); + UCollationResult(*strcoll) (const UCollator *coll, + const UChar *source, + int32_t sourceLength, + const UChar *target, + int32_t targetLength); + UCollationResult(*strcollUTF8) (const UCollator *coll, + const char *source, + int32_t sourceLength, + const char *target, + int32_t targetLength, + UErrorCode *status); + int32_t (*getSortKey) (const UCollator *coll, + const UChar *source, + int32_t sourceLength, + uint8_t *result, + int32_t resultLength); + int32_t (*nextSortKeyPart) (const UCollator *coll, + UCharIterator *iter, + uint32_t state[2], + uint8_t *dest, + int32_t count, + UErrorCode *status); + void (*setUTF8) (UCharIterator *iter, + const char *s, + int32_t length); + const char *(*errorName) (UErrorCode code); + int32_t (*strToUpper) (UChar *dest, + int32_t destCapacity, + const UChar *src, + int32_t srcLength, + const char *locale, + UErrorCode *pErrorCode); + int32_t (*strToLower) (UChar *dest, + int32_t destCapacity, + const UChar *src, + int32_t srcLength, + const char *locale, + UErrorCode *pErrorCode); + int32_t (*strToTitle) (UChar *dest, + int32_t destCapacity, + const UChar *src, + int32_t srcLength, + UBreakIterator *titleIter, + const char *locale, + UErrorCode *pErrorCode); + struct pg_icu_library *next; +} pg_icu_library; + +#endif /* * We define our own wrapper around locale_t so we can keep the same @@ -84,12 +151,18 @@ struct pg_locale_struct { const char *locale; UCollator *ucol; + pg_icu_library *lib; } icu; #endif int dummy; /* in case we have neither LOCALE_T nor ICU */ } info; }; +#ifdef USE_ICU +#define PG_ICU_LIB(x) ((x)->info.icu.lib) +#define PG_ICU_COL(x) ((x)->info.icu.ucol) +#endif + typedef struct pg_locale_struct *pg_locale_t; extern PGDLLIMPORT struct pg_locale_struct default_locale; diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index d9b839c979..0ccbbb711e 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -1101,6 +1101,7 @@ HeapTupleTableSlot HistControl HotStandbyState I32 +ICU_Convert_BI_Func ICU_Convert_Func ID INFIX @@ -2854,6 +2855,7 @@ TypeName U U32 U8 +UBreakIterator UChar UCharIterator UColAttribute @@ -3482,6 +3484,7 @@ pg_funcptr_t pg_gssinfo pg_hmac_ctx pg_hmac_errno +pg_icu_library pg_int64 pg_local_to_utf_combined pg_locale_t -- 2.30.2
