Re: Collation version tracking for macOS

2024-02-13 Thread Thomas Munro
On Tue, Feb 13, 2024 at 9:25 AM Jeff Davis wrote: > On Sun, 2024-02-11 at 22:04 +0530, Robert Haas wrote: > > "icu_multilib must be loaded via shared_preload_libraries. > > icu_multilib ignores any ICU library with a major version greater > > than > > that with which PostgreSQL was built." > > >

Re: Collation version tracking for macOS

2024-02-12 Thread Robert Haas
On Tue, Feb 13, 2024 at 1:55 AM Jeff Davis wrote: > Postgres can and does latch on to the version of ICU it was compiled > against. It's a normal shared library dependency. > > The problem is that databases -- and the file structures -- outlive a > particular version of Postgres. So if Postgres

Re: Collation version tracking for macOS

2024-02-12 Thread Jeff Davis
On Sun, 2024-02-11 at 22:04 +0530, Robert Haas wrote: > 1. Here's what we think your OS package manager is probably going to > do. > 2. That's going to interact with PostgreSQL in this way that I will > now describe. > 3. See, that sucks, because of the stuff I said above about needing > stable

Re: Collation version tracking for macOS

2024-02-11 Thread Robert Haas
On Sun, Feb 4, 2024 at 10:42 PM Jeff Davis wrote: > I'm hesitant to put much more work into it (e.g. new patches, etc.) > without more feedback. Your opinion would certainly be valuable -- for > instance, when reading the docs, can you imagine yourself actually > using this if you ran into a

Re: Collation version tracking for macOS

2024-02-04 Thread Jeff Davis
On Thu, 2024-02-01 at 15:58 -0500, Robert Haas wrote: > Not that I'm the most qualified person to have an opinion on this > topic, but did you intend to attach this stuff to this email, or is > it > somewhere else? The previous patch is here:

Re: Collation version tracking for macOS

2024-02-01 Thread Robert Haas
On Sun, Jan 21, 2024 at 1:58 PM Jeff Davis wrote: > I rendered the docs I wrote as an HTML page and attached it to this > thread, to make it easier for others to read and comment. It's > basically a tool for experts who are willing to devote effort to > managing their collations and ICU

Re: Collation version tracking for macOS

2024-02-01 Thread vignesh C
On Mon, 22 Jan 2024 at 00:28, Jeff Davis wrote: > > On Sat, 2024-01-20 at 07:40 +0530, vignesh C wrote: > > This thread has been idle for a year now, It has stalled after a lot > > of discussion. > > @Jeff Davis: Do you want to try to restart the discussion by posting > > an updated version and

Re: Collation version tracking for macOS

2024-01-21 Thread Jeff Davis
On Sat, 2024-01-20 at 07:40 +0530, vignesh C wrote: > This thread has been idle for a year now, It has stalled after a lot > of discussion. > @Jeff Davis: Do you want to try to restart the discussion by posting > an updated version and see what happens? Thank you for following up. Yes, I'd like

Re: Collation version tracking for macOS

2024-01-19 Thread vignesh C
On Sat, 21 Jan 2023 at 02:24, Jeff Davis wrote: > > On Thu, 2023-01-19 at 00:11 -0800, Jeff Davis wrote: > > Attached are a new set of patches, including a major enhancement: the > > icu_multilib contrib module. > > Attached rebased v8. > > [ It looks like my email client truncated the last email

Re: Collation version tracking for macOS

2023-01-10 Thread Peter Eisentraut
On 05.12.22 22:33, Thomas Munro wrote: On Tue, Dec 6, 2022 at 6:45 AM Joe Conway wrote: On 12/5/22 12:41, Jeff Davis wrote: On Mon, 2022-12-05 at 16:12 +1300, Thomas Munro wrote: 1. I think we should seriously consider provider = ICU63. I still think search-by-collversion is a little too

Re: Collation version tracking for macOS

2022-12-05 Thread Thomas Munro
On Tue, Dec 6, 2022 at 6:45 AM Joe Conway wrote: > On 12/5/22 12:41, Jeff Davis wrote: > > On Mon, 2022-12-05 at 16:12 +1300, Thomas Munro wrote: > >> 1. I think we should seriously consider provider = ICU63. I still > >> think search-by-collversion is a little too magical, even though it > >>

Re: Collation version tracking for macOS

2022-12-05 Thread Joe Conway
On 12/5/22 12:41, Jeff Davis wrote: On Mon, 2022-12-05 at 16:12 +1300, Thomas Munro wrote: 1.  I think we should seriously consider provider = ICU63.  I still think search-by-collversion is a little too magical, even though it clearly can be made to work.  Of the non-magical systems, I think

Re: Collation version tracking for macOS

2022-12-05 Thread Jeff Davis
On Mon, 2022-12-05 at 16:12 +1300, Thomas Munro wrote: > 1.  I think we should seriously consider provider = ICU63.  I still > think search-by-collversion is a little too magical, even though it > clearly can be made to work.  Of the non-magical systems, I think > encoding the choice of library

Re: Collation version tracking for macOS

2022-12-05 Thread Robert Haas
On Sun, Dec 4, 2022 at 10:12 PM Thomas Munro wrote: > My tentative votes are: > > 1. I think we should seriously consider provider = ICU63. I still > think search-by-collversion is a little too magical, even though it > clearly can be made to work. Of the non-magical systems, I think >

Re: Collation version tracking for macOS

2022-12-04 Thread Thomas Munro
On Tue, Nov 29, 2022 at 7:51 PM Jeff Davis wrote: > On Sat, 2022-11-26 at 18:27 +1300, Thomas Munro wrote: > > On Thu, Nov 24, 2022 at 5:48 PM Thomas Munro > > wrote: > > > On Thu, Nov 24, 2022 at 3:07 PM Jeff Davis > > > wrote: > > > > I'd vote for 1 on the grounds that it's easier to document

Re: Collation version tracking for macOS

2022-12-01 Thread Dagfinn Ilmari Mannsåker
Jeff Davis writes: > On Mon, 2022-11-28 at 19:36 -0800, Jeff Davis wrote: >> On Mon, 2022-11-28 at 21:57 -0500, Robert Haas wrote: >> > That is ... astonishingly bad. >> >> https://unicode-org.atlassian.net/browse/CLDR-16175 > > Oops, reported in CLDR instead of ICU. Moved to: > >

Re: Collation version tracking for macOS

2022-11-29 Thread Michael Paquier
On Wed, Nov 30, 2022 at 01:50:51PM +1300, Thomas Munro wrote: > The new character added to Version 12.1 is: > > U+32FF SQUARE ERA NAME REIWA > > Version 12.1 adds that single character to enable software to be > rapidly updated to support the new Japanese era name in calendrical > systems and

Re: Collation version tracking for macOS

2022-11-29 Thread Thomas Munro
On Wed, Nov 30, 2022 at 1:25 PM Jeff Davis wrote: > On Wed, 2022-11-30 at 10:52 +1300, Thomas Munro wrote: > > On Wed, Nov 30, 2022 at 8:38 AM Jeff Davis wrote: > > > On Tue, 2022-11-29 at 10:46 -0800, Jeff Davis wrote: > > > https://unicode-org.atlassian.net/browse/ICU-22216 > > > > I'm no

Re: Collation version tracking for macOS

2022-11-29 Thread Thomas Munro
On Wed, Nov 30, 2022 at 1:32 PM Jeff Davis wrote: > On Wed, 2022-11-30 at 10:29 +1300, Thomas Munro wrote: > > On Wed, Nov 30, 2022 at 9:59 AM Jeff Davis wrote: > > > Here's what I found for the 'ar' locale (firstminor/lastminor are > > > the > > > icu library versions,

Re: Collation version tracking for macOS

2022-11-29 Thread Jeff Davis
On Wed, 2022-11-30 at 10:29 +1300, Thomas Munro wrote: > On Wed, Nov 30, 2022 at 9:59 AM Jeff Davis wrote: > > Here's what I found for the 'ar' locale (firstminor/lastminor are > > the > > icu library versions, firstcollversion/lastcollversion are their > > respective collation versions for the

Re: Collation version tracking for macOS

2022-11-29 Thread Jeff Davis
On Wed, 2022-11-30 at 10:52 +1300, Thomas Munro wrote: > On Wed, Nov 30, 2022 at 8:38 AM Jeff Davis wrote: > > On Tue, 2022-11-29 at 10:46 -0800, Jeff Davis wrote: > > > One bit of weirdness is that I may have found another ICU > > > problem. > > > > Reported as: > > > >

Re: Collation version tracking for macOS

2022-11-29 Thread Thomas Munro
On Wed, Nov 30, 2022 at 8:38 AM Jeff Davis wrote: > On Tue, 2022-11-29 at 10:46 -0800, Jeff Davis wrote: > > One bit of weirdness is that I may have found another ICU problem. > > Reported as: > > https://unicode-org.atlassian.net/browse/ICU-22216 I'm no expert on loader/linker arcana but I have

Re: Collation version tracking for macOS

2022-11-29 Thread Jeff Davis
On Wed, 2022-11-30 at 09:00 +1300, Thomas Munro wrote: > I'm struggling to understand what's new about proposal #6. Perhaps it's just a slight variant; I'm not sure. It's not a complete proposal yet. The difference I had in mind is that it would treat the built-in ICU differently from what is

Re: Collation version tracking for macOS

2022-11-29 Thread Thomas Munro
On Wed, Nov 30, 2022 at 9:59 AM Jeff Davis wrote: > Here's what I found for the 'ar' locale (firstminor/lastminor are the > icu library versions, firstcollversion/lastcollversion are their > respective collation versions for the given locale): > > firstminor | lastminor | firstcollversion |

Re: Collation version tracking for macOS

2022-11-29 Thread Jeff Davis
On Wed, 2022-11-30 at 08:41 +1300, Thomas Munro wrote: > In terms of user experience, I think that might mean that users of > 'zh' who encounter warnings after a minor upgrade would therefore > really only have the options of REFRESHing and rebuilding, or > downgrading the package, because there's

Re: Collation version tracking for macOS

2022-11-29 Thread Jeff Davis
On Tue, 2022-11-29 at 14:34 -0500, Joe Conway wrote: > I understand that it is not easily done, but if the combination of > collprovider + collversion does not represent specific immutable > ordering behavior for a given locale Given the u_versionToString() bug, we know the version string could

Re: Collation version tracking for macOS

2022-11-29 Thread Thomas Munro
On Wed, Nov 30, 2022 at 8:52 AM Robert Haas wrote: > On Tue, Nov 29, 2022 at 1:59 PM Jeff Davis wrote: > > 6. Create a new concept of a "locked down collation" that points at > > some specific collation code (identified by some combination of library > > version and collation version or whatever

Re: Collation version tracking for macOS

2022-11-29 Thread Robert Haas
On Tue, Nov 29, 2022 at 1:59 PM Jeff Davis wrote: > 6. Create a new concept of a "locked down collation" that points at > some specific collation code (identified by some combination of library > version and collation version or whatever else can be used to identify > it). If a collation is

Re: Collation version tracking for macOS

2022-11-29 Thread Thomas Munro
On Wed, Nov 30, 2022 at 8:03 AM Jeff Davis wrote: > On Wed, 2022-11-30 at 07:18 +1300, Thomas Munro wrote: > > I think it also includes the CLDR version for *some* locales. From a > > quick look, that includes 'ar', 'ru', 'tr', 'zh'. Jeff, would you > > mind sharing the same table for one of

Re: Collation version tracking for macOS

2022-11-29 Thread Jeff Davis
On Tue, 2022-11-29 at 10:46 -0800, Jeff Davis wrote: > One bit of weirdness is that I may have found another ICU problem. Reported as: https://unicode-org.atlassian.net/browse/ICU-22216 -- Jeff Davis PostgreSQL Contributor Team - AWS

Re: Collation version tracking for macOS

2022-11-29 Thread Jeff Davis
On Mon, 2022-11-28 at 19:36 -0800, Jeff Davis wrote: > On Mon, 2022-11-28 at 21:57 -0500, Robert Haas wrote: > > That is ... astonishingly bad. > > https://unicode-org.atlassian.net/browse/CLDR-16175 Oops, reported in CLDR instead of ICU. Moved to:

Re: Collation version tracking for macOS

2022-11-29 Thread Joe Conway
On 11/29/22 13:59, Jeff Davis wrote: On Tue, 2022-11-29 at 11:27 -0500, Joe Conway wrote: My vote is for something like #5. The collversion should indicate a specific immutable ordering behavior. Easier said than done:

Re: Collation version tracking for macOS

2022-11-29 Thread Jeff Davis
On Wed, 2022-11-30 at 07:18 +1300, Thomas Munro wrote: > On Wed, Nov 30, 2022 at 7:03 AM Jeremy Schneider > wrote: > > It seems to me that the collator_version field is not a good > > version > > identifier to use. > > > > Just taking a quick glance at the ICU home page right now, it shows > >

Re: Collation version tracking for macOS

2022-11-29 Thread Jeff Davis
On Tue, 2022-11-29 at 11:27 -0500, Joe Conway wrote: > My vote is for something like #5. The collversion should indicate a > specific immutable ordering behavior. Easier said than done: https://www.postgresql.org/message-id/abddc35a7a447d93e2b8371a1a9052cb48866070.ca...@j-davis.com Even

Re: Collation version tracking for macOS

2022-11-29 Thread Jeff Davis
On Tue, 2022-11-29 at 12:32 -0500, Robert Haas wrote: > You know more about this than I do, for sure, so don't let my vote > back the project into a bad spot. I'm going back and forth myself. I haven't found a great answer here yet. > But, yeah, the thing you mention > here is what I'm worried

Re: Collation version tracking for macOS

2022-11-29 Thread Thomas Munro
On Wed, Nov 30, 2022 at 7:03 AM Jeremy Schneider wrote: > It seems to me that the collator_version field is not a good version > identifier to use. > > Just taking a quick glance at the ICU home page right now, it shows that > all of the last 5 versions of ICU have included "additions and >

Re: Collation version tracking for macOS

2022-11-29 Thread Jeremy Schneider
On 11/28/22 6:54 PM, Jeff Davis wrote: > > =# select * from pg_icu_collation_versions('en_US') order by > icu_version; > icu_version | uca_version | collator_version > -+-+-- > ... > 67.1| 13.0| 153.14 > 68.2| 13.0|

Re: Collation version tracking for macOS

2022-11-29 Thread Robert Haas
On Mon, Nov 28, 2022 at 11:49 PM Jeff Davis wrote: > Not necessarily, #2-4 (at least as implemented in v7) can only load one > major version at a time, so can't specify minor versions: > https://www.postgresql.org/message-id/9f8e9b5a3352478d4cf7d6c0a5dd7e82496be4b6.ca...@j-davis.com > > With #1,

Re: Collation version tracking for macOS

2022-11-29 Thread Joe Conway
On 11/28/22 14:11, Robert Haas wrote: On Wed, Nov 23, 2022 at 12:09 AM Thomas Munro wrote: OK. Time for a new list of the various models we've discussed so far: 1. search-by-collversion: We introduce no new "library version" concept to COLLATION and DATABASE object and little or no new

Re: Collation version tracking for macOS

2022-11-28 Thread Jeff Davis
On Sat, 2022-11-26 at 18:27 +1300, Thomas Munro wrote: > On Thu, Nov 24, 2022 at 5:48 PM Thomas Munro > wrote: > > On Thu, Nov 24, 2022 at 3:07 PM Jeff Davis > > wrote: > > > I'd vote for 1 on the grounds that it's easier to document and > > > understand a single collation version, which comes

Re: Collation version tracking for macOS

2022-11-28 Thread Jeff Davis
On Mon, 2022-11-28 at 14:11 -0500, Robert Haas wrote: > I don't really understand #1 or #5 well enough to have an educated > opinion, but I do think that #1 seems a bit magical. It hopes that > the > combination of a collation name and a datcollversion will be > sufficient to find exactly one

Re: Collation version tracking for macOS

2022-11-28 Thread Thomas Munro
On Tue, Nov 29, 2022 at 3:55 PM Jeff Davis wrote: > =# select * from pg_icu_collation_versions('en_US') order by > icu_version; > icu_version | uca_version | collator_version > -+-+-- > 50.2| 6.2 | 58.0.6.50 > 51.3| 6.2 |

Re: Collation version tracking for macOS

2022-11-28 Thread Jeff Davis
On Mon, 2022-11-28 at 21:57 -0500, Robert Haas wrote: > That is ... astonishingly bad. https://unicode-org.atlassian.net/browse/CLDR-16175 -- Jeff Davis PostgreSQL Contributor Team - AWS

Re: Collation version tracking for macOS

2022-11-28 Thread Robert Haas
On Mon, Nov 28, 2022 at 9:55 PM Jeff Davis wrote: > But did you notice that the version went backwards from 65.1 -> 66.1? > Well, actually, it didn't. The version of that collation in 66.1 went > from 153.97 -> 153.104. But there's a bug in versionToString() that > does the decimal output

Re: Collation version tracking for macOS

2022-11-28 Thread Jeff Davis
On Sat, 2022-11-26 at 18:27 +1300, Thomas Munro wrote: > Here's the first iteration. I will send a full review shortly, but I encountered an ICU bug along the way, which caused me some confusion for a bit. I'll skip past the various levels of confusion I had (burned a couple hours), and get right

Re: Collation version tracking for macOS

2022-11-28 Thread Robert Haas
On Wed, Nov 23, 2022 at 12:09 AM Thomas Munro wrote: > OK. Time for a new list of the various models we've discussed so far: > > 1. search-by-collversion: We introduce no new "library version" > concept to COLLATION and DATABASE object and little or no new syntax. > > 2.

Re: Collation version tracking for macOS

2022-11-27 Thread Thomas Munro
On Sat, Nov 26, 2022 at 6:27 PM Thomas Munro wrote: > This is just a first cut, but enough to try out and see if we like it, > what needs to be improved, what edge cases we haven't thought about > etc. Let me know what you think. BTW one problem to highlight (mentioned but buried in the test

Re: Collation version tracking for macOS

2022-11-25 Thread Thomas Munro
On Thu, Nov 24, 2022 at 5:48 PM Thomas Munro wrote: > On Thu, Nov 24, 2022 at 3:07 PM Jeff Davis wrote: > > I'd vote for 1 on the grounds that it's easier to document and > > understand a single collation version, which comes straight from > > ucol_getVersion(). This approach makes it a separate

Re: Collation version tracking for macOS

2022-11-23 Thread Thomas Munro
On Thu, Nov 24, 2022 at 3:07 PM Jeff Davis wrote: > I'd vote for 1 on the grounds that it's easier to document and > understand a single collation version, which comes straight from > ucol_getVersion(). This approach makes it a separate problem to find > the collation version among whatever

Re: Collation version tracking for macOS

2022-11-23 Thread Thomas Munro
On Thu, Nov 24, 2022 at 3:07 PM Jeff Davis wrote: > I'm sure this has been discussed, but which distros even support > multiple major versions of ICU? For Debian and friends, you can install any number of libicuNN packages (if you can find them eg from previous release repos), but there's only

Re: Collation version tracking for macOS

2022-11-23 Thread Jeff Davis
On Wed, 2022-11-23 at 18:08 +1300, Thomas Munro wrote: > (1) the default behaviour on failure to search would > likely be to use the linked library instead and WARN about > [dat]collversion mismatch, so far the same, and  Agreed. > (2) the set of people > who would really be prepared to compile

Re: Collation version tracking for macOS

2022-11-22 Thread Thomas Munro
On Tue, Nov 22, 2022 at 7:34 PM Jeff Davis wrote: > On Sat, 2022-10-22 at 14:22 +1300, Thomas Munro wrote: > > Problem 2: If ICU 67 ever decides to report a different version for > > a > > given collation (would it ever do that? I don't expect so, but ...), > > we'd be unable to open the

Re: Collation version tracking for macOS

2022-11-21 Thread Jeff Davis
On Sat, 2022-10-22 at 14:22 +1300, Thomas Munro wrote: > Problem 2: If ICU 67 ever decides to report a different version for > a > given collation (would it ever do that? I don't expect so, but ...), > we'd be unable to open the collation with the search-by-collversion > design, and

Re: Collation version tracking for macOS

2022-11-18 Thread Thomas Munro
On Sat, Nov 19, 2022 at 7:38 AM Thomas Munro wrote: > On Tue, Nov 15, 2022 at 1:55 PM Jeff Davis wrote: > > I realize your patch is experimental, but when there is a better > > consensus on the approach, we should consider adding declarative syntax > > such as: > > > >CREATE COLLATION (or

Re: Collation version tracking for macOS

2022-11-18 Thread Thomas Munro
Replying to Peter and Jeff in one email. On Sat, Nov 12, 2022 at 3:57 AM Peter Eisentraut wrote: > On 22.10.22 03:22, Thomas Munro wrote: > > I'd love to hear others' thoughts on how we can turn this into a > > workable solution. Hopefully while staying simple... > > I played with this patch a

Re: Collation version tracking for macOS

2022-11-14 Thread Jeff Davis
I looked at v6. * We'll need some clearer instructions on how to build/install extra ICU versions that might not be provided by the distribution packaging. For instance, I got a cryptic error until I used --enable-rpath, which might not be obvious to all users. * Can we have a better error

Re: Collation version tracking for macOS

2022-11-11 Thread Peter Eisentraut
On 22.10.22 03:22, Thomas Munro wrote: I'd love to hear others' thoughts on how we can turn this into a workable solution. Hopefully while staying simple... I played with this patch a bit. It looks like a reasonable approach. Attached is a small patch to get the dynamic libicu* lookup

Re: Collation version tracking for macOS

2022-11-08 Thread Thomas Munro
On Tue, Nov 8, 2022 at 1:22 AM Peter Eisentraut wrote: > I made a Homebrew repository for ICU versions 50 through 72: > https://github.com/petere/homebrew-icu Nice! > All of these packages build and pass their self-tests on my machine. So > from that experience, I think maintaining a

Re: Collation version tracking for macOS

2022-11-07 Thread Peter Eisentraut
On 02.11.22 00:57, Thomas Munro wrote: 3. Library availability. This is a problem for downstream communities to solve. For example, the people who build Windows installers might want to start bundling the ICU versions from their earlier releases, the people involved with each Linux/BSD distro

Re: Collation version tracking for macOS

2022-11-01 Thread Thomas Munro
On Wed, Nov 2, 2022 at 1:42 AM Thomas Munro wrote: > On Tue, Nov 1, 2022 at 11:33 PM Peter Eisentraut > wrote: > > What I'm wondering is where those ICU installations are going to come > > from. In order for this project to be viable, we would need to convince > > some combination of ICU

Re: Collation version tracking for macOS

2022-11-01 Thread Thomas Munro
On Tue, Nov 1, 2022 at 11:33 PM Peter Eisentraut wrote: > What I'm wondering is where those ICU installations are going to come > from. In order for this project to be viable, we would need to convince > some combination of ICU maintainers, OS packagers, and PGDG packagers to > provide and

Re: Collation version tracking for macOS

2022-11-01 Thread Peter Eisentraut
On 22.10.22 03:22, Thomas Munro wrote: Suppose your pgdata encounters a PostgreSQL linked against a later ICU library, most likely after an OS upgrade or migratoin, a pg_upgrade, or via streaming replication. You might get a new error "can't find ICU collation 'en' with version '153.14'; HINT:

Re: Collation version tracking for macOS

2022-10-21 Thread Thomas Munro
On Sat, Oct 22, 2022 at 10:24 AM Thomas Munro wrote: > ... But it > doesn't provide a way for me to create a new database that uses 63 on > purpose when I know what I'm doing. There are various reasons I might > want to do that. Thinking some more about this, I guess that could be addressed by

Re: Collation version tracking for macOS

2022-10-21 Thread Thomas Munro
Hi, Here is a rebase of this experimental patch. I think the basic mechanics are promising, but we haven't agreed on a UX. I hope we can figure this out. Restating the choice made in this branch of the experiment: Here I try to be just like DB2 (if I understood its manual correctly). In DB2,

Re: Collation version tracking for macOS

2022-06-14 Thread Peter Eisentraut
On 14.06.22 21:10, Jeremy Schneider wrote: Does Unicode CDLR provide (or even track) versioning of collation or other i18n functionality for individual locale settings? Yes. You can see that in PostgreSQL as various pre-seeded ICU collations having different versions.

Re: Collation version tracking for macOS

2022-06-14 Thread Jeremy Schneider
  > On Jun 14, 2022, at 19:06, Thomas Munro wrote: > One difference would be the effect if ICU ever ships a minor library > version update that changes the reported collversion. If I’m reading it correctly, ICU would not change collation in major versions, as an explicit matter of policy

Re: Collation version tracking for macOS

2022-06-14 Thread Thomas Munro
On Wed, Jun 15, 2022 at 7:10 AM Jeremy Schneider wrote: > > On Jun 14, 2022, at 14:10, Peter Eisentraut > > wrote: > > Conversely, why are we looking at the ICU version instead of the collation > > version. If we have recorded the collation as being version 1234, we need > > to look through

Re: Collation version tracking for macOS

2022-06-14 Thread Jeremy Schneider
> On Jun 14, 2022, at 14:10, Peter Eisentraut > wrote: >  > Conversely, why are we looking at the ICU version instead of the collation > version. If we have recorded the collation as being version 1234, we need to > look through the available ICU versions (assuming we can load multiple

Re: Collation version tracking for macOS

2022-06-14 Thread Peter Eisentraut
On 11.06.22 05:35, Peter Geoghegan wrote: Do we even need to store a version for indexes most of the time if we're versioning ICU itself, as part of the "time travelling collations" design? For that matter, do we even need to version collations directly anymore? Conversely, why are we looking

Re: Collation version tracking for macOS

2022-06-13 Thread Peter Geoghegan
On Mon, Jun 13, 2022 at 5:41 PM Thomas Munro wrote: > It'd clearly be a terrible idea for us to try to use any of that, and > Mac users should be very happy with the new support for ICU as DB > default. This suggests something that I already suspected: nobody particularly expects the system lib

Re: Collation version tracking for macOS

2022-06-13 Thread Thomas Munro
On Thu, Jun 9, 2022 at 11:33 AM Thomas Munro wrote: > On Thu, Jun 9, 2022 at 5:42 AM Tom Lane wrote: > > I'm sure that Apple are indeed updating the UTF8 data behind > > their proprietary i18n APIs, but the libc APIs are mostly getting benign > > neglect. > > As for how exactly they might be

Re: Collation version tracking for macOS

2022-06-12 Thread Thomas Munro
Hey Jeremy, On Tue, Jun 7, 2022 at 12:42 PM Jeremy Schneider wrote: > Thomas - thanks for the link back to one of the threads. I spent some time > reading through that and it’s a lot of material; I haven’t read the whole > thread yet. If you have some others that would also be particularly

Re: Collation version tracking for macOS

2022-06-11 Thread Thomas Munro
On Sun, Jun 12, 2022 at 11:59 AM Thomas Munro wrote: > > On Sat, Jun 11, 2022 at 4:21 PM Peter Geoghegan wrote: > > What about "time travel collations", but without the time travel part? > > That is, what about supporting multiple ICU versions per cluster, but > > not per database? So you could

Re: Collation version tracking for macOS

2022-06-11 Thread Thomas Munro
On Sat, Jun 11, 2022 at 4:21 PM Peter Geoghegan wrote: > What about "time travel collations", but without the time travel part? > That is, what about supporting multiple ICU versions per cluster, but > not per database? So you could upgrade the OS and Postgres, using > standard packages that

Re: Collation version tracking for macOS

2022-06-10 Thread Peter Geoghegan
On Fri, Jun 10, 2022 at 9:08 PM Thomas Munro wrote: > They're still useful for non-ICU collations (for example FreeBSD and > Windows can tell you about version changes based on open standards), > and they're *maybe* still useful for ICU, considering that there > are minor version upgrades, though

Re: Collation version tracking for macOS

2022-06-10 Thread Peter Geoghegan
On Fri, Jun 10, 2022 at 8:47 PM Thomas Munro wrote: > I'm also suspicious that there are more subtle hazards like pathkeys > lurking in the shadows. We go to great effort to recognise matching > and non-matching collations by OID alone, which is why my first > attempt was "distinct [OIDs]", so

Re: Collation version tracking for macOS

2022-06-10 Thread Thomas Munro
On Sat, Jun 11, 2022 at 3:36 PM Peter Geoghegan wrote: > Do we even need to store a version for indexes most of the time if > we're versioning ICU itself, as part of the "time travelling > collations" design? For that matter, do we even need to version > collations directly anymore? They're

Re: Collation version tracking for macOS

2022-06-10 Thread Thomas Munro
On Sat, Jun 11, 2022 at 2:29 PM Peter Geoghegan wrote: > The special REINDEX (or whatever) won't work as an atomic > operation...but that doesn't mean that the system as a whole will have > a mix of old and new physical collations forever, or even for very > long. So while everything still has to

Re: Collation version tracking for macOS

2022-06-10 Thread Peter Geoghegan
On Thu, Jun 9, 2022 at 9:31 PM Thomas Munro wrote: > Perhaps that could be modeled with a pg_depend row pointing to a > pg_icu_library row, which you'd probably need anyway, to prevent a > registered ICU library that is needed for a live index from being > dropped. (That's assuming that the

Re: Collation version tracking for macOS

2022-06-10 Thread Peter Geoghegan
On Fri, Jun 10, 2022 at 6:48 PM Thomas Munro wrote: > Executive summary of experiments so far: the "distinct collations" > concept is quite simple and robust, but exposes all the versions to > users and probably makes it really hard to upgrade (details not worked > out), while the "time

Re: Collation version tracking for macOS

2022-06-10 Thread Thomas Munro
On Fri, Jun 10, 2022 at 4:30 PM Thomas Munro wrote: > I'm not sold on any particular plan, but working through some examples > helped me see your idea better... I may try to code that up in a > minimal way so we can kick the tyres... I did a bit of hacking on that idea. The goal was to stamp

Re: Collation version tracking for macOS

2022-06-09 Thread Thomas Munro
On Fri, Jun 10, 2022 at 12:48 PM Tobias Bussmann wrote: > Perhaps I can shed some light on this matter: Hi Tobias, Oh, thanks for your answers. Definitely a few bits of interesting archeology I was not aware of. > Apple's libc collations have always been a bit special in that concern, even >

Re: Collation version tracking for macOS

2022-06-09 Thread Thomas Munro
On Fri, Jun 10, 2022 at 1:48 PM Peter Geoghegan wrote: > On Thu, Jun 9, 2022 at 6:23 PM Thomas Munro wrote: > > Well I can report that the system from ec483147 was hellishly > > complicated, and not universally loved. Which isn't to say that there > > isn't a simple and loveable way to do it,

Re: Collation version tracking for macOS

2022-06-09 Thread Peter Geoghegan
On Thu, Jun 9, 2022 at 6:23 PM Thomas Munro wrote: > Well I can report that the system from ec483147 was hellishly > complicated, and not universally loved. Which isn't to say that there > isn't a simple and loveable way to do it, waiting to be discovered, > and I do think we could fix most of

Re: Collation version tracking for macOS

2022-06-09 Thread Tobias Bussmann
Am 08.06.2022 um 16:16 schrieb Tom Lane : > The proposed patch would result in a warning about every collation- > sensitive index during every macOS major version upgrade, ie about > once a year for most people. > We need something that has at least *some* connection to actual changes. In

Re: Collation version tracking for macOS

2022-06-09 Thread Thomas Munro
On Fri, Jun 10, 2022 at 1:06 PM Peter Geoghegan wrote: > On Thu, Jun 9, 2022 at 5:59 PM Thomas Munro wrote: > > That sounds nice, but introduces subtle problems for the planner. For > > example, pathkeys that look compatible might not be, when > > merge-joining an ICU 63 index scan against an

Re: Collation version tracking for macOS

2022-06-09 Thread Peter Geoghegan
On Thu, Jun 9, 2022 at 5:59 PM Thomas Munro wrote: > That sounds nice, but introduces subtle problems for the planner. For > example, pathkeys that look compatible might not be, when > merge-joining an ICU 63 index scan against an ICU 67 index scan. You > could teach it about that, whereas with

Re: Collation version tracking for macOS

2022-06-09 Thread Thomas Munro
On Fri, Jun 10, 2022 at 12:32 PM Peter Geoghegan wrote: > On Thu, Jun 9, 2022 at 5:18 PM Thomas Munro wrote: > > You seem to have some > > other idea in mind where the system only knows about one > > "en-US-x-icu", but somehow, somewhere else (where?), keeps track of > > which indexes were built

Re: Collation version tracking for macOS

2022-06-09 Thread Tobias Bussmann
Thanks for picking this up! > How can I see evidence of this? I'm comparing Debian, FreeBSD and > macOS 12.4 and when I run "LC_COLLATE=en_US.UTF-8 sort > /usr/share/dict/words" I get upper and lower case mixed together on > the other OSes, but on the Mac the upper case comes first, which is my

Re: Collation version tracking for macOS

2022-06-09 Thread Peter Geoghegan
On Thu, Jun 9, 2022 at 5:18 PM Thomas Munro wrote: > However, since you mentioned that a simple REINDEX would get you from > one library version to another, I think we're making some completely > different assumptions somewhere along the line, and I don't get your > idea yet. It sounds like you

Re: Collation version tracking for macOS

2022-06-09 Thread Thomas Munro
On Fri, Jun 10, 2022 at 10:29 AM Peter Geoghegan wrote: > On Thu, Jun 9, 2022 at 2:20 PM Finnerty, Jim wrote: > > For example, an alternate syntax might be: > > > > create collation icu63."en-US-x-icu" (provider = icu, locale = > > 'en-US@colVersion=63'); > > Why would a user want to

Re: Collation version tracking for macOS

2022-06-09 Thread Peter Geoghegan
On Thu, Jun 9, 2022 at 4:23 PM Thomas Munro wrote: > Suppose you pg_upgrade to something that is linked against 71. > Perhaps you'd need to tell it how to dlopen 67 before you can open any > collations with that library, but once you've done that your > collation-dependent partition constraints

Re: Collation version tracking for macOS

2022-06-09 Thread Thomas Munro
On Fri, Jun 10, 2022 at 9:20 AM Finnerty, Jim wrote: > Specifying the library name before the language-country code with a new > separator (":") as you suggested below has some benefits. One of the reasons for putting some representation of desired library into the colliculocale column (rather

Re: Collation version tracking for macOS

2022-06-09 Thread Peter Geoghegan
On Thu, Jun 9, 2022 at 2:20 PM Finnerty, Jim wrote: > Specifying the library name before the language-country code with a new > separator (":") as you suggested below has some benefits. Did you consider > making the collation version just another collation attribute, such as > colStrength,

Re: Collation version tracking for macOS

2022-06-09 Thread Finnerty, Jim
Specifying the library name before the language-country code with a new separator (":") as you suggested below has some benefits. Did you consider making the collation version just another collation attribute, such as colStrength, colCaseLevel, etc.? For example, an alternate syntax might

Re: Collation version tracking for macOS

2022-06-09 Thread Peter Geoghegan
On Thu, Jun 9, 2022 at 10:54 AM Jeremy Schneider wrote: > MySQL did the right thing here by doing what every other RDBMS did, and just > making a simple “good-enough” collation hardcoded in the DB, same across all > platforms, that never changes. That's not true. Both SQL Server and DB2 have

Re: Collation version tracking for macOS

2022-06-09 Thread Peter Geoghegan
On Thu, Jun 9, 2022 at 10:54 AM Jeremy Schneider wrote: > I’m probably just going to end up rehashing the old threads I haven’t read > yet… > > One challenge with this approach is you have things like sort-merge joins > that require the same collation across multiple objects. So I think you’d

Re: Collation version tracking for macOS

2022-06-09 Thread Jeremy Schneider
> On Jun 8, 2022, at 22:40, Peter Geoghegan wrote: > > On Wed, Jun 8, 2022 at 10:24 PM Jeremy Schneider > wrote: >> Even if PG supports two versions of ICU, how does someone actually go about >> removing every dependency on the old version and replacing it with the new? > > They simply

Re: Collation version tracking for macOS

2022-06-09 Thread Peter Geoghegan
On Wed, Jun 8, 2022 at 10:39 PM Peter Geoghegan wrote: > They simply REINDEX, without changing anything. The details are still > fuzzy, but at least that's what I was thinking of. As I said before, BCP47 format tags are incredibly forgiving by design. So it should be reasonable to assume that

Re: Collation version tracking for macOS

2022-06-08 Thread Peter Geoghegan
On Wed, Jun 8, 2022 at 10:24 PM Jeremy Schneider wrote: > Even if PG supports two versions of ICU, how does someone actually go about > removing every dependency on the old version and replacing it with the new? They simply REINDEX, without changing anything. The details are still fuzzy, but at

  1   2   >