Re: PG 16 draft release notes ready
On Wed, Aug 23, 2023 at 09:36:01AM -0700, Jeff Davis wrote: > On Tue, 2023-08-22 at 22:23 -0400, Bruce Momjian wrote: > > > I notice that this item is still listed: > > > > > > * Determine the ICU default locale from the environment (Jeff > > > Davis) > > > > > > But that was reverted as part of 2535c74b1a. > > > > The original commit is: > > > > Author: Jeff Davis > > 2023-03-10 [c45dc7ffb] initdb: derive encoding from locale > > for ICU; similar to > > > > and I don't see that reverted by 2535c74b1a. Is that a problem? > > c45dc7ffb causes initdb to choose the encoding based on the environment > for ICU just like libc, and that was not reverted, so in v16: > > $ export LANG=en_US > $ initdb -D data --locale-provider=icu --icu-locale=en > ... > The default database encoding has accordingly been set to "LATIN1". > > Whereas previously in v15 that would cause an error like: > > initdb: error: encoding mismatch > initdb: detail: The encoding you selected (UTF8) and the encoding > that the selected locale uses (LATIN1) do not match... > > "Determine the ICU default locale from the environment" to me refers to > what happened in 27b62377b4, where initdb would select an ICU locale if > one was not provided. 2535c74b1a reverted that, so in v16: > > $ initdb -D data --locale-provider=icu > initdb: error: ICU locale must be specified > > Just like in v15. Okay, so what I hear you saying is that commit c45dc7ffb needs to remain in the release notes, but its description sounds like 27b62377b4, which was reverted, so my description is wrong for c45dc7ffb. I would love to blame the patch revert on this mistake, but looking at the history of this entry, I just didn't understand it when I initiallly wrote it. Updated applied patch attached. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you. diff --git a/doc/src/sgml/release-16.sgml b/doc/src/sgml/release-16.sgml index 47c72b..ddd8bc3f3b 100644 --- a/doc/src/sgml/release-16.sgml +++ b/doc/src/sgml/release-16.sgml @@ -1566,15 +1566,12 @@ Author: Jeff Davis -Determine the ICU default locale from the -environment (Jeff Davis) +Determine the default encoding from the locale when using +ICU (Jeff Davis) -However, ICU doesn't -support the C locale so UTF-8 is used in such -cases. Previously the default was always UTF-8. +Previously the default was always UTF-8.
Re: PG 16 draft release notes ready
On Tue, 2023-08-22 at 22:23 -0400, Bruce Momjian wrote: > > I notice that this item is still listed: > > > > * Determine the ICU default locale from the environment (Jeff > > Davis) > > > > But that was reverted as part of 2535c74b1a. > > The original commit is: > > Author: Jeff Davis > 2023-03-10 [c45dc7ffb] initdb: derive encoding from locale > for ICU; similar to > > and I don't see that reverted by 2535c74b1a. Is that a problem? c45dc7ffb causes initdb to choose the encoding based on the environment for ICU just like libc, and that was not reverted, so in v16: $ export LANG=en_US $ initdb -D data --locale-provider=icu --icu-locale=en ... The default database encoding has accordingly been set to "LATIN1". Whereas previously in v15 that would cause an error like: initdb: error: encoding mismatch initdb: detail: The encoding you selected (UTF8) and the encoding that the selected locale uses (LATIN1) do not match... "Determine the ICU default locale from the environment" to me refers to what happened in 27b62377b4, where initdb would select an ICU locale if one was not provided. 2535c74b1a reverted that, so in v16: $ initdb -D data --locale-provider=icu initdb: error: ICU locale must be specified Just like in v15. Regards, Jeff Davis
Re: PG 16 draft release notes ready
On Tue, Aug 22, 2023 at 01:42:41PM -0700, Jeff Davis wrote: > On Thu, 2023-05-18 at 16:49 -0400, Bruce Momjian wrote: > > I have completed the first draft of the PG 16 release notes. You can > > see the output here: > > > https://www.postgresql.org/docs/16/release-16.html#RELEASE-16-LOCALIZATION > > I notice that this item is still listed: > > * Determine the ICU default locale from the environment (Jeff Davis) > > But that was reverted as part of 2535c74b1a. The original commit is: Author: Jeff Davis 2023-03-10 [c45dc7ffb] initdb: derive encoding from locale for ICU; similar to and I don't see that reverted by 2535c74b1a. Is that a problem? -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: PG 16 draft release notes ready
On Thu, 2023-05-18 at 16:49 -0400, Bruce Momjian wrote: > I have completed the first draft of the PG 16 release notes. You can > see the output here: https://www.postgresql.org/docs/16/release-16.html#RELEASE-16-LOCALIZATION I notice that this item is still listed: * Determine the ICU default locale from the environment (Jeff Davis) But that was reverted as part of 2535c74b1a. Regards, Jeff Davis
Re: PG 16 draft release notes ready
On Tue, Aug 22, 2023 at 10:02:16AM +0300, Pavel Luzanov wrote: > On 22.08.2023 00:58, Bruce Momjian wrote: > > Attached is an applied patch that moves the inherit item into > > incompatibilities. clarifies it, and splits out the ADMIN syntax item. > > > The role's default inheritance behavior can be overridden with the new > GRANT ... WITH INHERIT clause. > > The only question about "can be". Why not "will be"? The inheritance > behavior will be changed anyway. I used "can be" to highlight you "can" override it, but don't need to. > > Please let me know if I need any other changes. Thanks. > > * Allow psql's access privilege commands to show system objects (Nathan > Bossart, Pavel Luzanov) > The options are \dpS, \zS, and \drg. > > I think that this description correct only for the \dpS and \zS commands. > (By the way, unfortunately after reverting MAINTAIN privilege this commands > are not much useful in v16.) > > But the \drg command is a different thing. This is a full featured > replacement for "Member of" column of the \du, \dg commands. > It shows not only members, but granted options (admin, inherit, set) and > grantor. > This is important information for membership usage and administration. > IMO, removing the "Member of" column from the \du & \dg commands also > requires attention in release notes. > > So, I suggest new item in the psql section for \drg. Something like this: > > * Add psql \drg command to display role grants and remove the "Member of" > column from \du & \dg altogether. I see your point. Attached is an applied patch which fixes this by splitting \drg into a separate item and adding text. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you. diff --git a/doc/src/sgml/release-16.sgml b/doc/src/sgml/release-16.sgml index 1f98ccdc62..c464be5ee1 100644 --- a/doc/src/sgml/release-16.sgml +++ b/doc/src/sgml/release-16.sgml @@ -2176,11 +2176,28 @@ Author: Tom Lane -Allow psql's access privilege commands to show system objects (Nathan Bossart, Pavel Luzanov) +Add psql command \drg to show role membership details (Pavel Luzanov) -The options are \dpS, \zS, and \drg. +The Member of output column has been removed from \du and \dg because this new command displays this informaion in more detail. + + + + + + + +Allow psql's access privilege commands to show system objects (Nathan Bossart) + + + +The options are \dpS and \zS.
Re: PG 16 draft release notes ready
On 22.08.2023 00:58, Bruce Momjian wrote: Attached is an applied patch that moves the inherit item into incompatibilities. clarifies it, and splits out the ADMIN syntax item. > The role's default inheritance behavior can be overridden with the new GRANT ... WITH INHERIT clause. The only question about "can be". Why not "will be"? The inheritance behavior will be changed anyway. Please let me know if I need any other changes. Thanks. * Allow psql's access privilege commands to show system objects (Nathan Bossart, Pavel Luzanov) The options are \dpS, \zS, and \drg. I think that this description correct only for the \dpS and \zS commands. (By the way, unfortunately after reverting MAINTAIN privilege this commands are not much useful in v16.) But the \drg command is a different thing. This is a full featured replacement for "Member of" column of the \du, \dg commands. It shows not only members, but granted options (admin, inherit, set) and grantor. This is important information for membership usage and administration. IMO, removing the "Member of" column from the \du & \dg commands also requires attention in release notes. So, I suggest new item in the psql section for \drg. Something like this: * Add psql \drg command to display role grants and remove the "Member of" column from \du & \dg altogether. -- Pavel Luzanov Postgres Professional: https://postgrespro.com
Re: PG 16 draft release notes ready
On Tue, 22 Aug 2023 at 10:08, Bruce Momjian wrote: > > On Sat, Aug 19, 2023 at 04:24:48AM +0200, Erwin Brandstetter wrote: > > I posted to pgsql-docs first, but was kindly redirected here by Jonathan: > > > > The release notes for Postgres 16 says here: > > https://www.postgresql.org/docs/16/release-16.html#RELEASE-16-PERFORMANCE > > > > Same as here: > > https://momjian.us/pgsql_docs/release-16.html#RELEASE-16-PERFORMANCE > > > > Allow window functions to use ROWS mode internally when RANGE mode is > > specified but unnecessary (David Rowley) > > Yes, I didn't like "specified" myself but never returned to improve it. > I am now using: > > Allow window functions to use the faster linkend="syntax-window-functions">ROWS mode > internally when RANGE mode is active but > unnecessary > -- > (David Rowley) > > Can that be improved? Looks good to me. > > Also, I was hoping to be mentioned in the release note for working this out: > > > > Allow window functions to use the faster ROWS mode internally when RANGE > > mode is specified or would be default, but unnecessary (David Rowley, Erwin > > Brandstetter) > > Uh, I have CC'ed David Rowley because that is unclear based on the > commit message. I don't normally mention reviewers. I confirm that Erwin reported in [1] that row_number() is not affected by the ROWS/RANGE option and that ROWS performs better due to the executor having less work to do. I am the author of the patch which implemented that plus a few other window functions that also can benefit from the same optimisation. Based on this, I don't see any problems with the credits for this item as they are currently in the release notes. David [1] https://postgr.es/m/CAGHENJ7LBBszxS%2BSkWWFVnBmOT2oVsBhDMB1DFrgerCeYa_DyA%40mail.gmail.com
Re: PG 16 draft release notes ready
On Sat, Aug 19, 2023 at 04:24:48AM +0200, Erwin Brandstetter wrote: > I posted to pgsql-docs first, but was kindly redirected here by Jonathan: > > The release notes for Postgres 16 says here: > https://www.postgresql.org/docs/16/release-16.html#RELEASE-16-PERFORMANCE > > Same as here: > https://momjian.us/pgsql_docs/release-16.html#RELEASE-16-PERFORMANCE > > Allow window functions to use ROWS mode internally when RANGE mode is > specified but unnecessary (David Rowley) Yes, I didn't like "specified" myself but never returned to improve it. I am now using: Allow window functions to use the faster ROWS mode internally when RANGE mode is active but unnecessary -- (David Rowley) Can that be improved? > But the improvement (fix to some degree) also applies to the much more common > case where no mode has been specified, RANGE unfortunately being the default. > That includes the most common use case "row_number() OVER (ORDER BY col)", > where RANGE mode should not be applied to begin with, according to SQL specs. > This is what made me investigate, test and eventually propose a fix in the > first place. See: Yes, very true. > Also, I was hoping to be mentioned in the release note for working this out: > > Allow window functions to use the faster ROWS mode internally when RANGE > mode is specified or would be default, but unnecessary (David Rowley, Erwin > Brandstetter) Uh, I have CC'ed David Rowley because that is unclear based on the commit message. I don't normally mention reviewers. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: PG 16 draft release notes ready
On Sat, Aug 19, 2023 at 12:59:47PM -0400, Bruce Momjian wrote: > On Thu, Aug 17, 2023 at 08:37:28AM +0300, Pavel Luzanov wrote: > > I can try to explain how I understand it myself. > > > > In v15 and early, inheritance of granted to role privileges depends on > > INHERIT attribute of a role: > > > > create user alice; > > grant pg_read_all_settings to alice; > > > > By default privileges inherited: > > \c - alice > > show data_directory; > > data_directory > > - > > /var/lib/postgresql/15/main > > (1 row) > > > > After disabling the INHERIT attribute, privileges are not inherited: > > > > \c - postgres > > alter role alice noinherit; > > > > \c - alice > > show data_directory; > > ERROR: must be superuser or have privileges of pg_read_all_settings to > > examine "data_directory" > > > > In v16 changing INHERIT attribute on alice role doesn't change inheritance > > behavior of already granted roles. > > If we repeat the example, Alice still inherits pg_read_all_settings > > privileges after disabling the INHERIT attribute for the role. > > > > Information for making decisions about role inheritance has been moved from > > the role attribute to GRANT role TO role [WITH INHERIT|NOINHERIT] command > > and can be viewed by the new \drg command: > > > > \drg > > List of role grants > > Role name | Member of | Options | Grantor > > ---+--+--+-- > > alice | pg_read_all_settings | INHERIT, SET | postgres > > (1 row) > > > > Changing the INHERIT attribute for a role now will affect (as the default > > value) only future GRANT commands without an INHERIT clause. > > I was able to create this simple example to illustrate it: > > CREATE ROLE a1; > CREATE ROLE a2; > CREATE ROLE a3; > CREATE ROLE a4; > CREATE ROLE b INHERIT; > > GRANT a1 TO b WITH INHERIT TRUE; > GRANT a2 TO b WITH INHERIT FALSE; > > GRANT a3 TO b; > ALTER USER b NOINHERIT; > GRANT a4 TO b; > > \drg > List of role grants >Role name | Member of | Options| Grantor > ---+---+--+-- >b | a1| INHERIT, SET | postgres >b | a2| SET | postgres >b | a3| INHERIT, SET | postgres >b | a4| SET | postgres > > I will work on the relase notes adjustments for this and reply in a few > days. Attached is an applied patch that moves the inherit item into incompatibilities. clarifies it, and splits out the ADMIN syntax item. Please let me know if I need any other changes. Thanks. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you. diff --git a/doc/src/sgml/release-16.sgml b/doc/src/sgml/release-16.sgml index c9c4fc07ca..c4ae566900 100644 --- a/doc/src/sgml/release-16.sgml +++ b/doc/src/sgml/release-16.sgml @@ -229,6 +229,24 @@ Collations and locales can vary between databases so having them as read-only se + + + + +Role inheritance now controls the default inheritance status of member roles added during GRANT (Robert Haas) + + + +The role's default inheritance behavior can be overridden with the new GRANT ... WITH INHERIT clause. +This allows inheritance of some roles and not others because the members' inheritance status is set at GRANT time. +Previously the inheritance status of member roles was controlled only by the role's inheritance status, and +changes to a role's inheritance status affected all previous and future member roles. + + +
Re: PG 16 draft release notes ready
On Thu, Aug 17, 2023 at 08:37:28AM +0300, Pavel Luzanov wrote: > On 17.08.2023 05:36, Bruce Momjian wrote: > > On Wed, Aug 9, 2023 at 08:35:21PM -0400, Bruce Momjian wrote: > > > On Sat, Aug 5, 2023 at 04:08:47PM -0700, Noah Misch wrote: > > > > > Author: Robert Haas > > > > > 2022-08-25 [e3ce2de09] Allow grant-level control of role inheritance > > > > > behavior. > > > > > --> > > > > > > > > > > > > > > > > > > > > Allow GRANT to control role inheritance behavior (Robert Haas) > > > > > > > > > > > > > > > > > > > > By default, role inheritance is controlled by the inheritance status > > > > > of the member role. The new GRANT clauses WITH INHERIT and WITH > > > > > ADMIN can now override this. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Allow roles that create other roles to automatically inherit the new > > > > > role's rights or SET ROLE to the new role (Robert Haas, Shi Yu) > > > > > > > > > > > > > > > > > > > > This is controlled by server variable createrole_self_grant. > > > > > > > > > > > > > > Similarly, v16 radically changes the CREATE ROLE ... WITH INHERIT > > > > clause. The > > > > clause used to "change the behavior of already-existing grants." Let's > > > > merge > > > > these two and move the combination to the incompatibilities section. > > > I need help with this. I don't understand how they can be combined, and > > > I don't understand the incompatibility text in commit e3ce2de09d: > > > > > > If a GRANT does not specify WITH INHERIT, the behavior based on > > > whether the member role is marked INHERIT or NOINHERIT. This means > > > that if all roles are marked INHERIT or NOINHERIT before any role > > > grants are performed, the behavior is identical to what we had > > > before; > > > otherwise, it's different, because ALTER ROLE [NO]INHERIT now only > > > changes the default behavior of future grants, and has no effect on > > > existing ones. > > I am waiting for an answer to this question, or can I assume the release > > notes are acceptable? > > I can try to explain how I understand it myself. > > In v15 and early, inheritance of granted to role privileges depends on > INHERIT attribute of a role: > > create user alice; > grant pg_read_all_settings to alice; > > By default privileges inherited: > \c - alice > show data_directory; > data_directory > - > /var/lib/postgresql/15/main > (1 row) > > After disabling the INHERIT attribute, privileges are not inherited: > > \c - postgres > alter role alice noinherit; > > \c - alice > show data_directory; > ERROR: must be superuser or have privileges of pg_read_all_settings to > examine "data_directory" > > In v16 changing INHERIT attribute on alice role doesn't change inheritance > behavior of already granted roles. > If we repeat the example, Alice still inherits pg_read_all_settings > privileges after disabling the INHERIT attribute for the role. > > Information for making decisions about role inheritance has been moved from > the role attribute to GRANT role TO role [WITH INHERIT|NOINHERIT] command > and can be viewed by the new \drg command: > > \drg > List of role grants > Role name | Member of | Options | Grantor > ---+--+--+-- > alice | pg_read_all_settings | INHERIT, SET | postgres > (1 row) > > Changing the INHERIT attribute for a role now will affect (as the default > value) only future GRANT commands without an INHERIT clause. I was able to create this simple example to illustrate it: CREATE ROLE a1; CREATE ROLE a2; CREATE ROLE a3; CREATE ROLE a4; CREATE ROLE b INHERIT; GRANT a1 TO b WITH INHERIT TRUE; GRANT a2 TO b WITH INHERIT FALSE; GRANT a3 TO b; ALTER USER b NOINHERIT; GRANT a4 TO b; \drg List of role grants Role name | Member of | Options| Grantor ---+---+--+-- b | a1| INHERIT, SET | postgres b | a2| SET | postgres b | a3| INHERIT, SET | postgres b | a4| SET | postgres I will work on the relase notes adjustments for this and reply in a few days. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: PG 16 draft release notes ready
I posted to pgsql-docs first, but was kindly redirected here by Jonathan: The release notes for Postgres 16 says here: https://www.postgresql.org/docs/16/release-16.html#RELEASE-16-PERFORMANCE Same as here: https://momjian.us/pgsql_docs/release-16.html#RELEASE-16-PERFORMANCE Allow window functions to use ROWS mode internally when RANGE mode is specified but unnecessary (David Rowley) But the improvement (fix to some degree) also applies to the much more common case where no mode has been specified, RANGE unfortunately being the default. That includes the most common use case "row_number() OVER (ORDER BY col)", where RANGE mode should not be applied to begin with, according to SQL specs. This is what made me investigate, test and eventually propose a fix in the first place. See: https://www.postgresql.org/message-id/flat/CAGHENJ7LBBszxS%2BSkWWFVnBmOT2oVsBhDMB1DFrgerCeYa_DyA%40mail.gmail.com https://www.postgresql.org/message-id/flat/CAApHDvohAKEtTXxq7Pc-ic2dKT8oZfbRKeEJP64M0B6%2BS88z%2BA%40mail.gmail.com Also, I was hoping to be mentioned in the release note for working this out: Allow window functions to use the faster ROWS mode internally when RANGE mode is specified or would be default, but unnecessary (David Rowley, Erwin Brandstetter) Thanks, Erwin On Sat, 19 Aug 2023 at 04:02, Bruce Momjian wrote: > I have completed the first draft of the PG 16 release notes. You can > see the output here: > > https://momjian.us/pgsql_docs/release-16.html > > I will adjust it to the feedback I receive; that URL will quickly show > all updates. > > I learned a few things creating it this time: > > * I can get confused over C function names and SQL function names in >commit messages. > > * The sections and ordering of the entries can greatly clarify the >items. > > * The feature count is slightly higher than recent releases: > > release-10: 189 > release-11: 170 > release-12: 180 > release-13: 178 > release-14: 220 > release-15: 184 > --> release-16: 200 > > -- > Bruce Momjian https://momjian.us > EDB https://enterprisedb.com > > Only you can decide what is important to you. > > > > >
Re: PG 16 draft release notes ready
On Wed, Aug 16, 2023 at 10:36:05PM -0400, Bruce Momjian wrote: > You can view the Postgres 16 release notes, with markup and links to our > docs, here: > > https://momjian.us/pgsql_docs/release-16.html FYI, thanks to this patch: commit 78ee60ed84 Author: Tom Lane Date: Mon Jan 9 15:08:24 2023 -0500 Doc: add XML ID attributes to and tags. This doesn't have any external effect at the moment, but it will allow adding useful link-discoverability features later. Brar Piening, reviewed by Karl Pinc. Discussion: https://postgr.es/m/CAB8KJ=jpuQU9QJe4+RgWENrK5g9jhoysMw2nvTN_esoOU0=a...@mail.gmail.com I was able to add more links to the docs, and the links were more precise. I used to be frustrated I couldn't find nearby links to content, but I had no such troubles this year. I think the additional and more precise links will help people digest the release notes more efficiently. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: PG 16 draft release notes ready
On 17.08.2023 05:36, Bruce Momjian wrote: On Wed, Aug 9, 2023 at 08:35:21PM -0400, Bruce Momjian wrote: On Sat, Aug 5, 2023 at 04:08:47PM -0700, Noah Misch wrote: Author: Robert Haas 2022-08-25 [e3ce2de09] Allow grant-level control of role inheritance behavior. --> Allow GRANT to control role inheritance behavior (Robert Haas) By default, role inheritance is controlled by the inheritance status of the member role. The new GRANT clauses WITH INHERIT and WITH ADMIN can now override this. Allow roles that create other roles to automatically inherit the new role's rights or SET ROLE to the new role (Robert Haas, Shi Yu) This is controlled by server variable createrole_self_grant. Similarly, v16 radically changes the CREATE ROLE ... WITH INHERIT clause. The clause used to "change the behavior of already-existing grants." Let's merge these two and move the combination to the incompatibilities section. I need help with this. I don't understand how they can be combined, and I don't understand the incompatibility text in commit e3ce2de09d: If a GRANT does not specify WITH INHERIT, the behavior based on whether the member role is marked INHERIT or NOINHERIT. This means that if all roles are marked INHERIT or NOINHERIT before any role grants are performed, the behavior is identical to what we had before; otherwise, it's different, because ALTER ROLE [NO]INHERIT now only changes the default behavior of future grants, and has no effect on existing ones. I am waiting for an answer to this question, or can I assume the release notes are acceptable? I can try to explain how I understand it myself. In v15 and early, inheritance of granted to role privileges depends on INHERIT attribute of a role: create user alice; grant pg_read_all_settings to alice; By default privileges inherited: \c - alice show data_directory; data_directory - /var/lib/postgresql/15/main (1 row) After disabling the INHERIT attribute, privileges are not inherited: \c - postgres alter role alice noinherit; \c - alice show data_directory; ERROR: must be superuser or have privileges of pg_read_all_settings to examine "data_directory" In v16 changing INHERIT attribute on alice role doesn't change inheritance behavior of already granted roles. If we repeat the example, Alice still inherits pg_read_all_settings privileges after disabling the INHERIT attribute for the role. Information for making decisions about role inheritance has been moved from the role attribute to GRANT role TO role [WITH INHERIT|NOINHERIT] command and can be viewed by the new \drg command: \drg List of role grants Role name | Member of | Options | Grantor ---+--+--+-- alice | pg_read_all_settings | INHERIT, SET | postgres (1 row) Changing the INHERIT attribute for a role now will affect (as the default value) only future GRANT commands without an INHERIT clause. -- Pavel Luzanov Postgres Professional: https://postgrespro.com
Re: PG 16 draft release notes ready
On Wed, Aug 9, 2023 at 08:35:21PM -0400, Bruce Momjian wrote: > On Sat, Aug 5, 2023 at 04:08:47PM -0700, Noah Misch wrote: > > > Author: Robert Haas > > > 2022-08-25 [e3ce2de09] Allow grant-level control of role inheritance > > > behavior. > > > --> > > > > > > > > > > > > Allow GRANT to control role inheritance behavior (Robert Haas) > > > > > > > > > > > > By default, role inheritance is controlled by the inheritance status of > > > the member role. The new GRANT clauses WITH INHERIT and WITH ADMIN can > > > now override this. > > > > > > > > > > > > > > > > > > > > > > > > Allow roles that create other roles to automatically inherit the new > > > role's rights or SET ROLE to the new role (Robert Haas, Shi Yu) > > > > > > > > > > > > This is controlled by server variable createrole_self_grant. > > > > > > > > > > Similarly, v16 radically changes the CREATE ROLE ... WITH INHERIT clause. > > The > > clause used to "change the behavior of already-existing grants." Let's > > merge > > these two and move the combination to the incompatibilities section. > > I need help with this. I don't understand how they can be combined, and > I don't understand the incompatibility text in commit e3ce2de09d: > > If a GRANT does not specify WITH INHERIT, the behavior based on > whether the member role is marked INHERIT or NOINHERIT. This means > that if all roles are marked INHERIT or NOINHERIT before any role > grants are performed, the behavior is identical to what we had before; > otherwise, it's different, because ALTER ROLE [NO]INHERIT now only > changes the default behavior of future grants, and has no effect on > existing ones. I am waiting for an answer to this question, or can I assume the release notes are acceptable? -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: PG 16 draft release notes ready
You can view the Postgres 16 release notes, with markup and links to our docs, here: https://momjian.us/pgsql_docs/release-16.html -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: PG 16 draft release notes ready
On Thu, Aug 10, 2023 at 07:56:12AM +0300, Pavel Luzanov wrote: > On 09.08.2023 21:06, Bruce Momjian wrote: > > On Sun, Jul 23, 2023 at 02:09:17PM +0300, Pavel Luzanov wrote: > > > Please consider to add item to the psql section: > > > > > > Add psql \drg command to display role grants and remove the "Member of" > > > column from \du & \dg altogether (d65ddaca) > > The release notes are only current as of 2023-06-26 and I will consider > > this when I updated them next week, thanks. > > This item is a part of Beta 3 scheduled for August 10, 2023 (today). [1] > It might be worth updating the release notes before the release. > But I'm not familiar with the release process in detail, so I could be > wrong. > > 1. > https://www.postgresql.org/message-id/93c00ac3-08b3-33ba-5d77-6ceb6ab20254%40postgresql.org The next text is: Allow psql's access privilege commands to show system objects (Nathan Bossart, Pavel Luzanov) --> The options are \dpS, \zS, and \drg. The current release notes are at: https://momjian.us/pgsql_docs/release-16.html -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: PG 16 draft release notes ready
On 09.08.2023 21:06, Bruce Momjian wrote: On Sun, Jul 23, 2023 at 02:09:17PM +0300, Pavel Luzanov wrote: Please consider to add item to the psql section: Add psql \drg command to display role grants and remove the "Member of" column from \du & \dg altogether (d65ddaca) The release notes are only current as of 2023-06-26 and I will consider this when I updated them next week, thanks. This item is a part of Beta 3 scheduled for August 10, 2023 (today). [1] It might be worth updating the release notes before the release. But I'm not familiar with the release process in detail, so I could be wrong. 1. https://www.postgresql.org/message-id/93c00ac3-08b3-33ba-5d77-6ceb6ab20254%40postgresql.org -- Pavel Luzanov Postgres Professional: https://postgrespro.com
Re: PG 16 draft release notes ready
FYI, the current PG 16 release notes are available at: https://momjian.us/pgsql_docs/release-16.html I plan to add markup next week. I am sorry I was away most of July so could not update this until now. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: PG 16 draft release notes ready
On Sat, Aug 5, 2023 at 04:08:47PM -0700, Noah Misch wrote: > > Remove libpq support for SCM credential authentication (Michael Paquier) > > Since the point of removing it is the deep unlikelihood of anyone using it, I > wouldn't list this in "incompatibilities". I moved this to the Source Code section. > > Deprecate createuser option --role (Nathan Bossart) > > This is indeed a deprecation, not a removal. By the definition of > "deprecate", it's not an incompatibility. I moved this to the Server Applications section. Thanks. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: PG 16 draft release notes ready
On Sat, Aug 5, 2023 at 04:08:47PM -0700, Noah Misch wrote: > > Author: Robert Haas > > 2022-08-25 [e3ce2de09] Allow grant-level control of role inheritance > > behavior. > > --> > > > > > > > > Allow GRANT to control role inheritance behavior (Robert Haas) > > > > > > > > By default, role inheritance is controlled by the inheritance status of the > > member role. The new GRANT clauses WITH INHERIT and WITH ADMIN can now > > override this. > > > > > > > > > > > > > > > > Allow roles that create other roles to automatically inherit the new role's > > rights or SET ROLE to the new role (Robert Haas, Shi Yu) > > > > > > > > This is controlled by server variable createrole_self_grant. > > > > > > Similarly, v16 radically changes the CREATE ROLE ... WITH INHERIT clause. The > clause used to "change the behavior of already-existing grants." Let's merge > these two and move the combination to the incompatibilities section. I need help with this. I don't understand how they can be combined, and I don't understand the incompatibility text in commit e3ce2de09d: If a GRANT does not specify WITH INHERIT, the behavior based on whether the member role is marked INHERIT or NOINHERIT. This means that if all roles are marked INHERIT or NOINHERIT before any role grants are performed, the behavior is identical to what we had before; otherwise, it's different, because ALTER ROLE [NO]INHERIT now only changes the default behavior of future grants, and has no effect on existing ones. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: PG 16 draft release notes ready
On Wed, Aug 09, 2023 at 05:45:27PM -0400, Bruce Momjian wrote: > That is very helpful. I added this to the release notes Server > Configuration section: > > > > Allow huge pages to work on newer versions of Windows 10 (Thomas Munro) > > > > This adds the special handling required to enable huge pages on newer > versions of Windows 10. > > Looks good to me, thanks for updating the notes! -- Michael signature.asc Description: PGP signature
Re: PG 16 draft release notes ready
On Sat, Aug 5, 2023 at 04:08:47PM -0700, Noah Misch wrote: > On Thu, May 18, 2023 at 04:49:47PM -0400, Bruce Momjian wrote: > > https://momjian.us/pgsql_docs/release-16.html > > > > > > > > > > > Restrict the privileges of CREATEROLE roles (Robert Haas) > > > > > > > > Previously roles with CREATEROLE privileges could change many aspects of > > any non-superuser role. Such changes, including adding members, now > > require the role requesting the change to have ADMIN OPTION > > permission. > > > > > > > > > > > > > > > > Improve logic of CREATEROLE roles ability to control other roles (Robert > > Haas) > > > > > > > > For example, they can change the CREATEDB, REPLICATION, and BYPASSRLS > > properties only if they also have those permissions. > > > > > > CREATEROLE is a radically different feature in v16. In v15-, it was an > almost-superuser. In v16, informally speaking, it can create and administer > its own collection of roles, but it can't administer roles outside its > collection or grant memberships or permissions not offered to itself. Hence, > let's move these two into the incompatibilities section. Let's also merge > them, since f1358ca52 is just doing to clauses like CREATEDB what cf5eb37c5 > did to role memberships. Good point. I have adjusted this item with the attached patch. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you. diff --git a/doc/src/sgml/release-16.sgml b/doc/src/sgml/release-16.sgml index 1213f876f4..cccdc01d11 100644 --- a/doc/src/sgml/release-16.sgml +++ b/doc/src/sgml/release-16.sgml @@ -244,6 +244,24 @@ Collations and locales can vary between databases so having them as read-only se + + + + +Restrict the privileges of CREATEROLE and its ability to modify other roles (Robert Haas) + + + +Previously roles with CREATEROLE privileges could change many aspects of any non-superuser role. Such changes, including adding members, now require the role requesting the change to have ADMIN OPTION +permission. For example, they can now change the CREATEDB, REPLICATION, and BYPASSRLS properties only if they also have those permissions. + + + - - - -Restrict the privileges of CREATEROLE roles (Robert Haas) - - - -Previously roles with CREATEROLE privileges could change many aspects of any non-superuser role. Such changes, including adding members, now require the role requesting the change to have ADMIN OPTION -permission. - - - - - - - -Improve logic of CREATEROLE roles ability to control other roles (Robert Haas) - - - -For example, they can change the CREATEDB, REPLICATION, and BYPASSRLS properties only if they also have those permissions. - - -
Re: PG 16 draft release notes ready
On Sun, Jul 23, 2023 at 08:19:55PM +0900, Michael Paquier wrote: > On Tue, Jul 04, 2023 at 05:32:07PM -0400, Bruce Momjian wrote: > > On Tue, Jul 4, 2023 at 03:31:05PM +0900, Michael Paquier wrote: > >> On Thu, May 18, 2023 at 04:49:47PM -0400, Bruce Momjian wrote: > >> Sawada-san has mentioned on twitter that fdd8937 is not mentioned in > >> the release notes, and it seems to me that he is right. This is > >> described as a bug in the commit log, but it did not get backpatched > >> because of the lack of complaints. Also, because we've removed > >> support for anything older than Windows 10 in PG16, this change very > >> easy to do. > > > > I did review this and wasn't sure exactly what I would describe. It is > > saying huge pages will now work on some versions of Windows 10 but > > didn't before? > > Windows 10 has always used a forced automated rolling upgrade process, > so there are not many versions older than 1703, I suppose. I don't > know if large pages were working before 1703 where > FILE_MAP_LARGE_PAGES has been introduced, and I have never been able > to test that. Honestly, I don't think that we need to be picky about > the version mentioned, as per the forced upgrade process done by > Microsoft. > > So, my preference would be to keep it simple and add an item like "Fix > huge pages on Windows 10 and newer versions", with as potential > subnote "The backend sets a flag named FILE_MAP_LARGE_PAGES to allow > huge pages", though this is not really mandatory to go down to this > level of internals, either. That is very helpful. I added this to the release notes Server Configuration section: Allow huge pages to work on newer versions of Windows 10 (Thomas Munro) This adds the special handling required to enable huge pages on newer versions of Windows 10. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: PG 16 draft release notes ready
On Sun, Jul 23, 2023 at 02:09:17PM +0300, Pavel Luzanov wrote: > Please consider to add item to the psql section: > > Add psql \drg command to display role grants and remove the "Member of" > column from \du & \dg altogether (d65ddaca) The release notes are only current as of 2023-06-26 and I will consider this when I updated them next week, thanks. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: PG 16 draft release notes ready
On Sun, Jul 23, 2023 at 12:45:17PM +0800, jian he wrote: > > https://momjian.us/pgsql_docs/release-16.html > > > > I will adjust it to the feedback I receive; that URL will quickly show > > all updates. > > > Create a predefined role and grantable privilege with permission to > > perform maintenance operations (Nathan Bossart) > > The predefined role is is called pg_maintain. > > this feature was also reverted. > https://git.postgresql.org/cgit/postgresql.git/commit/?id=151c22deee66a3390ca9a1c3675e29de54ae73fc Thenks, fixed based on earlier report by Laurenz Albe. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: PG 16 draft release notes ready
On Fri, Jul 14, 2023 at 09:29:42PM +0200, Erik Rijkers wrote: > Op 7/4/23 om 23:32 schreef Bruce Momjian: > > > > https://momjian.us/pgsql_docs/release-16.html > > I noticed these: > > 'new new RULES' should be > 'new RULES' > > 'Perform apply of large transactions' should be > 'Performs apply of large transactions' > (I think) > > 'SQL JSON paths' should be > 'SQL/JSON paths' Fixed with the attached patch. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you. diff --git a/doc/src/sgml/release-16.sgml b/doc/src/sgml/release-16.sgml index d8e1369844..c84c0f0eda 100644 --- a/doc/src/sgml/release-16.sgml +++ b/doc/src/sgml/release-16.sgml @@ -1370,7 +1370,7 @@ Allow custom ICU collation rules to be created (Peter Eisentraut) -This is done using CREATE COLLATION's new new RULES clause, as well as new options for CREATE DATABASE, createdb, and initdb. +This is done using CREATE COLLATION's new RULES clause, as well as new options for CREATE DATABASE, createdb, and initdb. @@ -1471,7 +1471,7 @@ Allow parallel application of logical replication (Hou Zhijie, Wang Wei, Amit Ka -The CREATE SUBSCRIPTION "streaming" option now supports "parallel" to enable parallel application. Perform apply of large transactions by parallel workers. The number of parallel workers is controlled by +The CREATE SUBSCRIPTION "streaming" option now supports "parallel" to enable application of large transactions by parallel workers. The number of parallel workers is controlled by the new server variable max_parallel_apply_workers_per_subscription. Wait events LogicalParallelApplyMain, LogicalParallelApplyStateChange, and LogicalApplySendData were also added. Column leader_pid was added to system view pg_stat_subscription to track parallel activity. @@ -1803,7 +1803,7 @@ Author: Peter Eisentraut -Add support for enhanced numeric literals in SQL JSON paths (Peter Eisentraut) +Add support for enhanced numeric literals in SQL/JSON paths (Peter Eisentraut)
Re: PG 16 draft release notes ready
On Fri, Jul 14, 2023 at 08:20:59PM +0200, Laurenz Albe wrote: > On Thu, 2023-05-18 at 16:49 -0400, Bruce Momjian wrote: > > I have completed the first draft of the PG 16 release notes. > > The release notes still have: > > - Have initdb use ICU by default if ICU is enabled in the binary (Jeff Davis) > > Option --locale-provider=libc can be used to disable ICU. > > > But this was reverted in 2535c74b1a6190cc42e13f6b6b55d94bff4b7dd6. FYI, this was corrected in this commit: commit c729642bd7 Author: Bruce Momjian Date: Fri Jun 30 17:35:47 2023 -0400 doc: PG 16 relnotes, remove "Have initdb use ICU by default" Item reverted. Backpatch-through: 16 only -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: PG 16 draft release notes ready
On Fri, Jul 14, 2023 at 08:16:38PM +0200, Laurenz Albe wrote: > On Thu, 2023-05-18 at 16:49 -0400, Bruce Momjian wrote: > > I have completed the first draft of the PG 16 release notes. > > The release notes say: > > - Prevent \df+ from showing function source code (Isaac Morland) > > Function bodies are more easily viewed with \ev and \ef. > > > That should be \sf, not \ev or \ef, right? Agreed, fixed. I am not sure why I put \ev and \ef there. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: PG 16 draft release notes ready
On Thu, May 18, 2023 at 04:49:47PM -0400, Bruce Momjian wrote: > https://momjian.us/pgsql_docs/release-16.html > > > > > Restrict the privileges of CREATEROLE roles (Robert Haas) > > > > Previously roles with CREATEROLE privileges could change many aspects of any > non-superuser role. Such changes, including adding members, now require the > role requesting the change to have ADMIN OPTION > permission. > > > > > > > > Improve logic of CREATEROLE roles ability to control other roles (Robert Haas) > > > > For example, they can change the CREATEDB, REPLICATION, and BYPASSRLS > properties only if they also have those permissions. > > CREATEROLE is a radically different feature in v16. In v15-, it was an almost-superuser. In v16, informally speaking, it can create and administer its own collection of roles, but it can't administer roles outside its collection or grant memberships or permissions not offered to itself. Hence, let's move these two into the incompatibilities section. Let's also merge them, since f1358ca52 is just doing to clauses like CREATEDB what cf5eb37c5 did to role memberships. > > > > > Allow GRANT to control role inheritance behavior (Robert Haas) > > > > By default, role inheritance is controlled by the inheritance status of the > member role. The new GRANT clauses WITH INHERIT and WITH ADMIN can now > override this. > > > > > > > > Allow roles that create other roles to automatically inherit the new role's > rights or SET ROLE to the new role (Robert Haas, Shi Yu) > > > > This is controlled by server variable createrole_self_grant. > > Similarly, v16 radically changes the CREATE ROLE ... WITH INHERIT clause. The clause used to "change the behavior of already-existing grants." Let's merge these two and move the combination to the incompatibilities section. > Remove libpq support for SCM credential authentication (Michael Paquier) Since the point of removing it is the deep unlikelihood of anyone using it, I wouldn't list this in "incompatibilities". > Deprecate createuser option --role (Nathan Bossart) This is indeed a deprecation, not a removal. By the definition of "deprecate", it's not an incompatibility.
Re: PG 16 draft release notes ready
On Tue, Jul 04, 2023 at 05:32:07PM -0400, Bruce Momjian wrote: > On Tue, Jul 4, 2023 at 03:31:05PM +0900, Michael Paquier wrote: >> On Thu, May 18, 2023 at 04:49:47PM -0400, Bruce Momjian wrote: >> Sawada-san has mentioned on twitter that fdd8937 is not mentioned in >> the release notes, and it seems to me that he is right. This is >> described as a bug in the commit log, but it did not get backpatched >> because of the lack of complaints. Also, because we've removed >> support for anything older than Windows 10 in PG16, this change very >> easy to do. > > I did review this and wasn't sure exactly what I would describe. It is > saying huge pages will now work on some versions of Windows 10 but > didn't before? Windows 10 has always used a forced automated rolling upgrade process, so there are not many versions older than 1703, I suppose. I don't know if large pages were working before 1703 where FILE_MAP_LARGE_PAGES has been introduced, and I have never been able to test that. Honestly, I don't think that we need to be picky about the version mentioned, as per the forced upgrade process done by Microsoft. So, my preference would be to keep it simple and add an item like "Fix huge pages on Windows 10 and newer versions", with as potential subnote "The backend sets a flag named FILE_MAP_LARGE_PAGES to allow huge pages", though this is not really mandatory to go down to this level of internals, either. -- Michael signature.asc Description: PGP signature
Re: PG 16 draft release notes ready
Please consider to add item to the psql section: Add psql \drg command to display role grants and remove the "Member of" column from \du & \dg altogether (d65ddaca) -- Pavel Luzanov Postgres Professional: https://postgrespro.com
Re: PG 16 draft release notes ready
> https://momjian.us/pgsql_docs/release-16.html > > I will adjust it to the feedback I receive; that URL will quickly show > all updates. > Create a predefined role and grantable privilege with permission to perform > maintenance operations (Nathan Bossart) > The predefined role is is called pg_maintain. this feature was also reverted. https://git.postgresql.org/cgit/postgresql.git/commit/?id=151c22deee66a3390ca9a1c3675e29de54ae73fc
Re: PG 16 draft release notes ready
Op 7/4/23 om 23:32 schreef Bruce Momjian: https://momjian.us/pgsql_docs/release-16.html I noticed these: 'new new RULES' should be 'new RULES' 'Perform apply of large transactions' should be 'Performs apply of large transactions' (I think) 'SQL JSON paths' should be 'SQL/JSON paths' Erik Rijkers
Re: PG 16 draft release notes ready
On Thu, 2023-05-18 at 16:49 -0400, Bruce Momjian wrote: > I have completed the first draft of the PG 16 release notes. The release notes still have: - Have initdb use ICU by default if ICU is enabled in the binary (Jeff Davis) Option --locale-provider=libc can be used to disable ICU. But this was reverted in 2535c74b1a6190cc42e13f6b6b55d94bff4b7dd6. Yours, Laurenz Albe
Re: PG 16 draft release notes ready
On Thu, 2023-05-18 at 16:49 -0400, Bruce Momjian wrote: > I have completed the first draft of the PG 16 release notes. The release notes say: - Prevent \df+ from showing function source code (Isaac Morland) Function bodies are more easily viewed with \ev and \ef. That should be \sf, not \ev or \ef, right? Yours, Laurenz Albe
Re: PG 16 draft release notes ready
On Tue, Jul 4, 2023 at 03:31:05PM +0900, Michael Paquier wrote: > On Thu, May 18, 2023 at 04:49:47PM -0400, Bruce Momjian wrote: > > I have completed the first draft of the PG 16 release notes. You can > > see the output here: > > > > https://momjian.us/pgsql_docs/release-16.html > > > > I will adjust it to the feedback I receive; that URL will quickly show > > all updates. > > Sawada-san has mentioned on twitter that fdd8937 is not mentioned in > the release notes, and it seems to me that he is right. This is > described as a bug in the commit log, but it did not get backpatched > because of the lack of complaints. Also, because we've removed > support for anything older than Windows 10 in PG16, this change very > easy to do. I did review this and wasn't sure exactly what I would describe. It is saying huge pages will now work on some versions of Windows 10 but didn't before? -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: PG 16 draft release notes ready
On Thu, May 18, 2023 at 04:49:47PM -0400, Bruce Momjian wrote: > I have completed the first draft of the PG 16 release notes. You can > see the output here: > > https://momjian.us/pgsql_docs/release-16.html > > I will adjust it to the feedback I receive; that URL will quickly show > all updates. Sawada-san has mentioned on twitter that fdd8937 is not mentioned in the release notes, and it seems to me that he is right. This is described as a bug in the commit log, but it did not get backpatched because of the lack of complaints. Also, because we've removed support for anything older than Windows 10 in PG16, this change very easy to do. -- Michael signature.asc Description: PGP signature
Re: PG 16 draft release notes ready
On Fri, Jun 30, 2023 at 05:29:17PM +0900, Masahiro Ikeda wrote: > Hi, > > Thanks for making the release notes. I found the release note of > PG16 beta2 mentions a reverted following feature. > > ``` > > > > > Have initdb use ICU by default if ICU is enabled in the binary (Jeff Davis) > > > > Option --locale-provider=libc can be used to disable ICU. > > > ``` > > Unfortunately, the feature is reverted with the commit. > * > https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=2535c74b1a6190cc42e13f6b6b55d94bff4b7dd6 Oh, I didn't notice the revert --- item removed. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: PG 16 draft release notes ready
Hi, Thanks for making the release notes. I found the release note of PG16 beta2 mentions a reverted following feature. ``` Have initdb use ICU by default if ICU is enabled in the binary (Jeff Davis) Option --locale-provider=libc can be used to disable ICU. ``` Unfortunately, the feature is reverted with the commit. * https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=2535c74b1a6190cc42e13f6b6b55d94bff4b7dd6 Regards, -- Masahiro Ikeda NTT DATA CORPORATION
Re: PG 16 draft release notes ready
On Tue, Jun 27, 2023 at 03:49:44PM -0600, Roberto Mello wrote: > Adding to this thread as suggested by jkatz for consideration of > adding to release notes... > > In [1] I mention the omission of ldap_password_hook and a suggested paragraph. > > Roberto > > [1] > https://www.postgresql.org/message-id/CAKz%3D%3DbLzGb-9O294AoZHqEWpAi2Ki58yCr4gaqg1HnZyh3L1uA%40mail.gmail.com I did see that commit: commit 419a8dd814 Author: Andrew Dunstan Date: Wed Mar 15 16:37:28 2023 -0400 Add a hook for modifying the ldapbind password The hook can be installed by a shared_preload library. A similar mechanism could be used for radius paswords, for example, and the type name auth_password_hook_typ has been shosen with that in mind. John Naylor and Andrew Dunstan Discussion: https://postgr.es/m/469b06ed-69de-ba59-c13a-91d2372e5...@dunslane.net However, there is no user documentation of this hook, so it didn't seem like something to add to the release notes. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: PG 16 draft release notes ready
Adding to this thread as suggested by jkatz for consideration of adding to release notes... In [1] I mention the omission of ldap_password_hook and a suggested paragraph. Roberto [1] https://www.postgresql.org/message-id/CAKz%3D%3DbLzGb-9O294AoZHqEWpAi2Ki58yCr4gaqg1HnZyh3L1uA%40mail.gmail.com
Re: PG 16 draft release notes ready
On Thu, Jun 8, 2023 at 02:23:33PM +0900, Masahiko Sawada wrote: > Hi, > > On Fri, May 19, 2023 at 5:49 AM Bruce Momjian wrote: > > > > I have completed the first draft of the PG 16 release notes. You can > > see the output here: > > > > https://momjian.us/pgsql_docs/release-16.html > > > > I will adjust it to the feedback I receive; that URL will quickly show > > all updates. > > > > > > > > Remove pg_walinspect functions > pg_get_wal_records_info_till_end_of_wal() and > pg_get_wal_stats_till_end_of_wal(). > > > > I found that this item misses the author, Bharath Rupireddy. Please > add his name. Thanks, fixed. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: PG 16 draft release notes ready
On Mon, 5 Jun 2023 11:42:43 -0400 Bruce Momjian wrote: > On Mon, Jun 5, 2023 at 05:33:51PM +0900, Yugo NAGATA wrote: > > Hello, > > > > On Thu, 18 May 2023 16:49:47 -0400 > > Bruce Momjian wrote: > > > > > I have completed the first draft of the PG 16 release notes. You can > > > see the output here: > > > > Thanks for the release notes. > > > > > > > > https://momjian.us/pgsql_docs/release-16.html > > > > > > I will adjust it to the feedback I receive; that URL will quickly show > > > all updates. > > > > I didn't find the following in the release note. This might be > > considered as a bug fix, but the change in this commit can potentially > > impact applications. Is it worth including it in the release note? > > > > commit 43351557d0d2b9c5e20298b5fee2849abef86aff > > Make materialized views participate in predicate locking > > I did look at this commit and decided, thought it is a behavior change, > that it is probably something that would be caught during upgrade > testing. I thought it was rare enough, and so hard to describe about > how to adjust for it, that is should not be mentioned. If we find that > users do hit this issue, or others, during beta, we can add it. Thank you for replying. I understood. Regards, Yugo Nagata > > -- > Bruce Momjian https://momjian.us > EDB https://enterprisedb.com > > Only you can decide what is important to you. -- Yugo NAGATA
Re: PG 16 draft release notes ready
Hi, On Fri, May 19, 2023 at 5:49 AM Bruce Momjian wrote: > > I have completed the first draft of the PG 16 release notes. You can > see the output here: > > https://momjian.us/pgsql_docs/release-16.html > > I will adjust it to the feedback I receive; that URL will quickly show > all updates. > Remove pg_walinspect functions pg_get_wal_records_info_till_end_of_wal() and pg_get_wal_stats_till_end_of_wal(). I found that this item misses the author, Bharath Rupireddy. Please add his name. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com
Re: PG 16 draft release notes ready
On Mon, Jun 5, 2023 at 05:33:51PM +0900, Yugo NAGATA wrote: > Hello, > > On Thu, 18 May 2023 16:49:47 -0400 > Bruce Momjian wrote: > > > I have completed the first draft of the PG 16 release notes. You can > > see the output here: > > Thanks for the release notes. > > > > > https://momjian.us/pgsql_docs/release-16.html > > > > I will adjust it to the feedback I receive; that URL will quickly show > > all updates. > > I didn't find the following in the release note. This might be > considered as a bug fix, but the change in this commit can potentially > impact applications. Is it worth including it in the release note? > > commit 43351557d0d2b9c5e20298b5fee2849abef86aff > Make materialized views participate in predicate locking I did look at this commit and decided, thought it is a behavior change, that it is probably something that would be caught during upgrade testing. I thought it was rare enough, and so hard to describe about how to adjust for it, that is should not be mentioned. If we find that users do hit this issue, or others, during beta, we can add it. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: PG 16 draft release notes ready
Hello, On Thu, 18 May 2023 16:49:47 -0400 Bruce Momjian wrote: > I have completed the first draft of the PG 16 release notes. You can > see the output here: Thanks for the release notes. > > https://momjian.us/pgsql_docs/release-16.html > > I will adjust it to the feedback I receive; that URL will quickly show > all updates. I didn't find the following in the release note. This might be considered as a bug fix, but the change in this commit can potentially impact applications. Is it worth including it in the release note? commit 43351557d0d2b9c5e20298b5fee2849abef86aff Make materialized views participate in predicate locking Regards, Yugo Nagata > I learned a few things creating it this time: > > * I can get confused over C function names and SQL function names in >commit messages. > > * The sections and ordering of the entries can greatly clarify the >items. > > * The feature count is slightly higher than recent releases: > > release-10: 189 > release-11: 170 > release-12: 180 > release-13: 178 > release-14: 220 > release-15: 184 > --> release-16: 200 > > -- > Bruce Momjian https://momjian.us > EDB https://enterprisedb.com > > Only you can decide what is important to you. > > -- Yugo NAGATA
Re: PG 16 draft release notes ready
On Wed, May 31, 2023 at 06:03:01PM +1200, David Rowley wrote: > I don't think this should go under "E.1.3.11. Source Code". The patch > was entirely aimed to increase performance, not just of allocations > themselves, but of any operations which uses palloc'd memory. This is > due to the patch increasing the density of memory allocation on blocks > malloc'd by our memory context code so that fewer CPU cache lines need > to be touched in the entire backend process for *all* memory that's > allocated with palloc. The performance increase here can be fairly > significant for small-sized palloc requests when CPU cache pressure is > high. Since CPU caches aren't that big, it does not take much of a > query to put the cache pressure up. Hashing or sorting a few million > rows is going to do that. > > The patch here was born out of the regression report I made in [1], > which I mention in [2] about the prototype patch Andres wrote to fix > the performance regression. > > I think "E.1.3.1.2. General Performance" might be a better location. > Having it under "Source Code" makes it sound like it was some > refactoring work. That's certainly not the case. Okay, moved. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: PG 16 draft release notes ready
On Wed, 31 May 2023 at 11:32, Bruce Momjian wrote: > > On Thu, May 25, 2023 at 05:57:25PM +1200, David Rowley wrote: > > On 64-bit builds, it was 16 bytes for AllocSet contexts, 24 bytes for > > generation contexts and 16 bytes for slab contexts. > > Okay, item added to Source Code: I don't think this should go under "E.1.3.11. Source Code". The patch was entirely aimed to increase performance, not just of allocations themselves, but of any operations which uses palloc'd memory. This is due to the patch increasing the density of memory allocation on blocks malloc'd by our memory context code so that fewer CPU cache lines need to be touched in the entire backend process for *all* memory that's allocated with palloc. The performance increase here can be fairly significant for small-sized palloc requests when CPU cache pressure is high. Since CPU caches aren't that big, it does not take much of a query to put the cache pressure up. Hashing or sorting a few million rows is going to do that. The patch here was born out of the regression report I made in [1], which I mention in [2] about the prototype patch Andres wrote to fix the performance regression. I think "E.1.3.1.2. General Performance" might be a better location. Having it under "Source Code" makes it sound like it was some refactoring work. That's certainly not the case. A bit more detail: Here's a small histogram of the number of allocations in various size buckets from running make check with some debug output in AllocSetAlloc and GenerationAlloc to record the size of the allocation: bucket | number_of_allocations | percent_of_total_allocations +---+- up to 16 bytes | 8,881,106 | 31.39 up to 32 bytes | 4,579,608 | 16.18 up to 64 bytes | 6,574,107 | 23.23 above 64 bytes | 8,260,714 | 29.19 So quite a large portion of our allocations (at least in our test suite) are small. Halving the 16-byte chunk header down 8 bytes on a 16-byte allocation means a 25% memory saving. David [1] https://www.postgresql.org/message-id/CAApHDvqXpLzav6dUeR5vO_RBh_feHrHMLhigVQXw9jHCyKP9PA%40mail.gmail.com [2] https://www.postgresql.org/message-id/CAApHDvowHNSVLhMc0cnovg8PfnYQZxit-gP_bn3xkT4rZX3G0w%40mail.gmail.com
Re: PG 16 draft release notes ready
On Thu, May 25, 2023 at 05:57:25PM +1200, David Rowley wrote: > On Thu, 25 May 2023 at 05:45, Bruce Momjian wrote: > > > > On Wed, May 24, 2023 at 01:43:50PM -0400, Bruce Momjian wrote: > > > > * Reduce palloc() memory overhead for all memory allocations down to 8 > > > > bytes on all platforms. (Andres Freund, David Rowley) > > > > > > > > This allows more efficient use of memory and is especially useful in > > > > queries which perform operations (such as sorting or hashing) that > > > > require more than work_mem. > > > > > > Well, this would go in the source code section, but it seems too > > > internal and global to mention. > > > > What was the previous memory allocation overhead? > > On 64-bit builds, it was 16 bytes for AllocSet contexts, 24 bytes for > generation contexts and 16 bytes for slab contexts. Okay, item added to Source Code: Reduce overhead of memory allocations (Andres Freund, David Rowley) -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: PG 16 draft release notes ready
On Tue, May 30, 2023 at 06:33:09AM -0400, Masahiko Sawada wrote: > Hi, > > On Thu, May 18, 2023 at 4:49 PM Bruce Momjian wrote: > > > > I have completed the first draft of the PG 16 release notes. You can > > see the output here: > > > > I have one suggestion on this item: > > > > > > Allow logical replication subscribers to process only changes that > have no origin (Vignesh C, Amit Kapila) > > > > This can be used to avoid replication loops. > > > > I think it's better to mention the new 'origin' option as other new > subscription options are mentioned. For example, > > > This can be used to avoid replication loops. This can be controlled by > the subscription "origin" option. > Great, new text is: This can be used to avoid replication loops. This is controlled by the new CREATE SUBSCRIPTION "origin" option. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: PG 16 draft release notes ready
Hi, On Thu, May 18, 2023 at 4:49 PM Bruce Momjian wrote: > > I have completed the first draft of the PG 16 release notes. You can > see the output here: > I have one suggestion on this item: Allow logical replication subscribers to process only changes that have no origin (Vignesh C, Amit Kapila) This can be used to avoid replication loops. I think it's better to mention the new 'origin' option as other new subscription options are mentioned. For example, This can be used to avoid replication loops. This can be controlled by the subscription "origin" option. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com
Re: PG 16 draft release notes ready
Bruce Momjian writes: > On Sat, May 27, 2023 at 09:34:37PM -0400, Bruce Momjian wrote: >> > > This is controlled by auto_explain.log_parameter_max_length, and by >> > > default query parameters will be logged with no length >> > > restriction. SHOULD THIS BE MORE CLEARLY IDENTIFIED AS CONTROLLING THE >> > > EXECUTION OF PREPARED STATEMENTS? >> > >> > This is wrong, the logging applies to all query parameters, not just for >> > prepared statements (and has nothing to do with controlling the >> > execution thereof). That was just the only way to test it when it was >> > written, because psql's \bind command exist yet then. >> >> I see your point. How is this? >> >> Allow auto_explain to log query parameters used by parameterized >> statements (Dagfinn Ilmari Mannsåker) >> >> This affects queries using server-side PRAPARE/EXECUTE >> and client-side parse/bind. Logging is controlled by >> auto_explain.log_parameter_max_length; by default query >> parameters will be logged with no length restriction. > > Done, attached patch applied. That works for me. Thanks! - ilmari
Re: PG 16 draft release notes ready
On Sat, May 27, 2023 at 09:34:37PM -0400, Bruce Momjian wrote: > > > This is controlled by auto_explain.log_parameter_max_length, and by > > > default query parameters will be logged with no length > > > restriction. SHOULD THIS BE MORE CLEARLY IDENTIFIED AS CONTROLLING THE > > > EXECUTION OF PREPARED STATEMENTS? > > > > This is wrong, the logging applies to all query parameters, not just for > > prepared statements (and has nothing to do with controlling the > > execution thereof). That was just the only way to test it when it was > > written, because psql's \bind command exist yet then. > > I see your point. How is this? > > Allow auto_explain to log query parameters used by parameterized > statements (Dagfinn Ilmari Mannsåker) > > This affects queries using server-side PRAPARE/EXECUTE > and client-side parse/bind. Logging is controlled by > auto_explain.log_parameter_max_length; by default query > parameters will be logged with no length restriction. Done, attached patch applied. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you. diff --git a/doc/src/sgml/release-16.sgml b/doc/src/sgml/release-16.sgml index 816286575f..32678074d2 100644 --- a/doc/src/sgml/release-16.sgml +++ b/doc/src/sgml/release-16.sgml @@ -2838,12 +2838,12 @@ Author: Michael Paquier -Allow auto_explain to log query parameters used in executing prepared statements (Dagfinn Ilmari Mannsåker) +Allow auto_explain to log values passed to parameterized statements (Dagfinn Ilmari Mannsåker) -This is controlled by auto_explain.log_parameter_max_length, and by default query parameters will be logged with no length restriction. -SHOULD THIS BE MORE CLEARLY IDENTIFIED AS CONTROLLING THE EXECUTION OF PREPARED STATEMENTS? +This affects queries using server-side PRAPARE/EXECUTE and client-side parse/bind. Logging is controlled by auto_explain.log_parameter_max_length; by default query parameters will +be logged with no length restriction.
Re: PG 16 draft release notes ready
On Wed, May 24, 2023 at 04:13:14PM +1200, David Rowley wrote: > On Wed, 24 May 2023 at 15:54, Bruce Momjian wrote: > > First, I need to mention parallel _hash_ join. Second, I think this > > item is saying that the _inner_ side of a parallel hash join can be an > > OUTER or FULL join. How about? > > > > Allow hash joins to be parallelized where the inner side is > > processed as an OUTER or FULL join (Melanie Plageman, Thomas Munro) > > > > In this case, the inner side is the hashed side. > > I think Jonathan's text is safe to swap OUTER to RIGHT as it mentions > "execution". For the release notes, maybe the mention of it can be > moved away from "E.1.3.1.1. Optimizer" and put under "E.1.3.1.2. > General Performance" and ensure we mention that we're talking about > the executor? > > I'm thinking it might be confusing if we claim that this is something > that we switched on in the planner. It was a limitation with the > executor which the planner was just onboard with not producing plans > for. Well, I try to keep plan changes in the optimizer section because that is where the decisions are made, and how people think of plans since EXPLAIN makes them visible. I agree it is an executor change but I think that distinction will be more confusing than helpful. Frankly, almost all the optimizer items are really executor changes. Maybe the "Optimizer" title needs to be changed, but I do think it is good to group plan changes together. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: PG 16 draft release notes ready
On Tue, May 23, 2023 at 11:54:30PM -0400, Bruce Momjian wrote: > On Wed, May 24, 2023 at 08:37:45AM +1200, David Rowley wrote: > > On Mon, 22 May 2023 at 07:05, Jonathan S. Katz wrote: > > > * Parallel execution of queries that use `FULL` and `OUTER` joins > > > > I think this should be `RIGHT` joins rather than `OUTER` joins. > > > > LEFT joins have been parallelizable I think for a long time now. > > Well, since we can swap left/right easily, why would we not have just > have swappted the tables and done the join in the past? I think there > are two things missing in my description. > > First, I need to mention parallel _hash_ join. Second, I think this > item is saying that the _inner_ side of a parallel hash join can be an > OUTER or FULL join. How about? > > Allow hash joins to be parallelized where the inner side is > processed as an OUTER or FULL join (Melanie Plageman, Thomas Munro) > > In this case, the inner side is the hashed side. I went with this text: Allow parallelization of FULL and internal right OUTER hash joins (Melanie Plageman, Thomas Munro) -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: PG 16 draft release notes ready
On Mon, May 29, 2023 at 10:08:41AM -0400, Masahiko Sawada wrote: > On Sun, May 21, 2023 at 10:47 PM Bruce Momjian wrote: > > > > Okay new merged item is: > > > > > > > > > > > > Allow logical decoding on standbys (Bertrand Drouvot, Andres > > Freund, Amit Khandekar, Bertrand Drouvot) > > > > > > > > > > > > New function pg_log_standby_snapshot() forces creation of WAL > > snapshots. > > Snapshots are required for logical slot creation so this function > > speeds their creation on standbys. > > > > > > > > Bertrand Drouvot is mentioned two times in this item and commit > 0fdab27ad is listed two times. Is it intentional? Thanks, fixed. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: PG 16 draft release notes ready
On Sun, May 21, 2023 at 10:47 PM Bruce Momjian wrote: > > Okay new merged item is: > > > > > > Allow logical decoding on standbys (Bertrand Drouvot, Andres Freund, > Amit Khandekar, Bertrand Drouvot) > > > > > > New function pg_log_standby_snapshot() forces creation of WAL > snapshots. > Snapshots are required for logical slot creation so this function > speeds their creation on standbys. > > > Bertrand Drouvot is mentioned two times in this item and commit 0fdab27ad is listed two times. Is it intentional? Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com
Re: PG 16 draft release notes ready
On Fri, May 26, 2023 at 12:21:23PM +0200, Álvaro Herrera wrote: > On 2023-May-25, Laurenz Albe wrote: > > > @@ -1335,7 +1335,7 @@ Author: Peter Eisentraut > > > > > > > > -Add Windows process the system collations (Jose Santamaria Flecha) > > +Add Windows to process the system collations (Jose Santamaria Flecha) > > ADD THIS? > > > > > > Hmm, not sure this describes the change properly. Maybe something like > "On Windows, system locales are now imported automatically. Previously, > only ICU locales were imported automatically on Windows." > > Maybe the Windows improvements should be listed together in a separate > section. > > Also, "Juan José Santamaría Flecha" is the spelling Juan José uses for > his name. Okay, I reword this and fixed Juan's name, attached, and applied. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you. diff --git a/doc/src/sgml/release-16.sgml b/doc/src/sgml/release-16.sgml index faecae7c42..1db72eeef3 100644 --- a/doc/src/sgml/release-16.sgml +++ b/doc/src/sgml/release-16.sgml @@ -1294,7 +1294,7 @@ Determine the ICU default locale from the environment (Jeff Davis) -However, ICU doesn't support the C local so UTF-8 is used in such cases. Previously the default was always UTF-8. +However, ICU doesn't support the C locale so UTF-8 is used in such cases. Previously the default was always UTF-8. @@ -1335,8 +1335,11 @@ Author: Peter Eisentraut -Add Windows process the system collations (Jose Santamaria Flecha) -ADD THIS? +Allow Windows to import system locales automatically (Juan José Santamaría Flecha) + + + +Previously, only ICU locales could be imported on Windows.
Re: PG 16 draft release notes ready
On Thu, May 25, 2023 at 11:51:24PM +0200, Laurenz Albe wrote: > On Thu, 2023-05-18 at 16:49 -0400, Bruce Momjian wrote: > > I have completed the first draft of the PG 16 release notes. > > I found two typos. > diff --git a/doc/src/sgml/release-16.sgml b/doc/src/sgml/release-16.sgml > index faecae7c42..7dad0b8550 100644 > --- a/doc/src/sgml/release-16.sgml > +++ b/doc/src/sgml/release-16.sgml > @@ -1294,7 +1294,7 @@ Determine the ICU default locale from the environment > (Jeff Davis) > > > > -However, ICU doesn't support the C local so UTF-8 is used in such cases. > Previously the default was always UTF-8. > +However, ICU doesn't support the C locale so UTF-8 is used in such cases. > Previously the default was always UTF-8. > > I have made this change. > @@ -1335,7 +1335,7 @@ Author: Peter Eisentraut > > > > -Add Windows process the system collations (Jose Santamaria Flecha) > +Add Windows to process the system collations (Jose Santamaria Flecha) > ADD THIS? > > I will deal with this item in the email from Álvaro Herrera. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: PG 16 draft release notes ready
On Thu, May 25, 2023 at 09:20:11PM +0100, Dagfinn Ilmari Mannsåker wrote: > Bruce Momjian writes: > > > I have completed the first draft of the PG 16 release notes. You can > > see the output here: > > > > https://momjian.us/pgsql_docs/release-16.html > > > > I will adjust it to the feedback I receive; that URL will quickly show > > all updates. > > The bit about auto_explain and query parameters says: > > > Allow auto_explain to log query parameters used in executing prepared > > statements (Dagfinn Ilmari Mannsåker) > > > > This is controlled by auto_explain.log_parameter_max_length, and by > > default query parameters will be logged with no length > > restriction. SHOULD THIS BE MORE CLEARLY IDENTIFIED AS CONTROLLING THE > > EXECUTION OF PREPARED STATEMENTS? > > This is wrong, the logging applies to all query parameters, not just for > prepared statements (and has nothing to do with controlling the > execution thereof). That was just the only way to test it when it was > written, because psql's \bind command exist yet then. I see your point. How is this? Allow auto_explain to log query parameters used by parameterized statements (Dagfinn Ilmari Mannsåker) This affects queries using server-side PRAPARE/EXECUTE and client-side parse/bind. Logging is controlled by auto_explain.log_parameter_max_length; by default query parameters will be logged with no length restriction. > Should we perhaps add some tests for that, like the attached? Sorry, I don't know the answer. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: PG 16 draft release notes ready
On 2023-May-25, Laurenz Albe wrote: > @@ -1335,7 +1335,7 @@ Author: Peter Eisentraut > > > > -Add Windows process the system collations (Jose Santamaria Flecha) > +Add Windows to process the system collations (Jose Santamaria Flecha) > ADD THIS? > > Hmm, not sure this describes the change properly. Maybe something like "On Windows, system locales are now imported automatically. Previously, only ICU locales were imported automatically on Windows." Maybe the Windows improvements should be listed together in a separate section. Also, "Juan José Santamaría Flecha" is the spelling Juan José uses for his name. -- Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/
Re: PG 16 draft release notes ready
On Thu, 2023-05-18 at 16:49 -0400, Bruce Momjian wrote: > I have completed the first draft of the PG 16 release notes. I found two typos. Yours, Laurenz Albe diff --git a/doc/src/sgml/release-16.sgml b/doc/src/sgml/release-16.sgml index faecae7c42..7dad0b8550 100644 --- a/doc/src/sgml/release-16.sgml +++ b/doc/src/sgml/release-16.sgml @@ -1294,7 +1294,7 @@ Determine the ICU default locale from the environment (Jeff Davis) -However, ICU doesn't support the C local so UTF-8 is used in such cases. Previously the default was always UTF-8. +However, ICU doesn't support the C locale so UTF-8 is used in such cases. Previously the default was always UTF-8. @@ -1335,7 +1335,7 @@ Author: Peter Eisentraut -Add Windows process the system collations (Jose Santamaria Flecha) +Add Windows to process the system collations (Jose Santamaria Flecha) ADD THIS?
Re: PG 16 draft release notes ready
Bruce Momjian writes:
> I have completed the first draft of the PG 16 release notes. You can
> see the output here:
>
> https://momjian.us/pgsql_docs/release-16.html
>
> I will adjust it to the feedback I receive; that URL will quickly show
> all updates.
The bit about auto_explain and query parameters says:
> Allow auto_explain to log query parameters used in executing prepared
> statements (Dagfinn Ilmari Mannsåker)
>
> This is controlled by auto_explain.log_parameter_max_length, and by
> default query parameters will be logged with no length
> restriction. SHOULD THIS BE MORE CLEARLY IDENTIFIED AS CONTROLLING THE
> EXECUTION OF PREPARED STATEMENTS?
This is wrong, the logging applies to all query parameters, not just for
prepared statements (and has nothing to do with controlling the
execution thereof). That was just the only way to test it when it was
written, because psql's \bind command exist yet then.
Should we perhaps add some tests for that, like the attached?
- ilmari
>From d3630f299fc2d2d9f9eb3addd426f98e5196100d Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Dagfinn=20Ilmari=20Manns=C3=A5ker?=
Date: Thu, 25 May 2023 21:13:11 +0100
Subject: [PATCH] Test auto_explain parameter logging with protocol-level bind
parameters
When auto_explain.log_parameter_max_length was added, psql didn't have
the \bind command for extended query protocol yet, so the test could
only use prepared statements.
---
contrib/auto_explain/t/001_auto_explain.pl | 31 ++
1 file changed, 31 insertions(+)
diff --git a/contrib/auto_explain/t/001_auto_explain.pl b/contrib/auto_explain/t/001_auto_explain.pl
index abb422f8de..d2a0078546 100644
--- a/contrib/auto_explain/t/001_auto_explain.pl
+++ b/contrib/auto_explain/t/001_auto_explain.pl
@@ -106,6 +106,21 @@ sub query_log
qr/Query Parameters:/,
"query parameters not logged when disabled, text mode");
+# bind parameters
+$log_contents = query_log($node,
+ q{SELECT * FROM pg_proc WHERE proname = $1 AND prokind = $2 \bind ascii f \g}
+);
+
+like(
+ $log_contents,
+ qr/Query Text: SELECT \* FROM pg_proc WHERE proname = \$1 AND prokind = \$2/,
+ "query text with parameters logged, text mode");
+
+like(
+ $log_contents,
+ qr/Query Parameters: \$1 = 'ascii', \$2 = 'f'/,
+ "query parameters logged, text mode");
+
# Query Identifier.
# Logging enabled.
$log_contents = query_log(
@@ -172,6 +187,22 @@ sub query_log
qr/"Node Type": "Index Scan"[^}]*"Index Name": "pg_class_relname_nsp_index"/s,
"index scan logged, json mode");
+# query with bind parameters in JSON format.
+$log_contents = query_log(
+ $node,
+ q{SELECT * FROM pg_class WHERE relname = $1 AND relkind = $2 \bind pg_proc r \x},
+ { "auto_explain.log_format" => "json" });
+
+like(
+ $log_contents,
+ qr/"Query Text": "SELECT \* FROM pg_class WHERE relname = \$1 AND relkind = \$2 "/,
+ "query text with parameters logged, json mode");
+
+like(
+ $log_contents,
+ qr/"Query Parameters": "\$1 = 'pg_proc', \$2 = 'r'"/,
+ "query parameters logged, json mode");
+
# Check that PGC_SUSET parameters can be set by non-superuser if granted,
# otherwise not
--
2.39.2
Re: PG 16 draft release notes ready
On Thu, 25 May 2023 at 05:45, Bruce Momjian wrote: > > On Wed, May 24, 2023 at 01:43:50PM -0400, Bruce Momjian wrote: > > > * Reduce palloc() memory overhead for all memory allocations down to 8 > > > bytes on all platforms. (Andres Freund, David Rowley) > > > > > > This allows more efficient use of memory and is especially useful in > > > queries which perform operations (such as sorting or hashing) that > > > require more than work_mem. > > > > Well, this would go in the source code section, but it seems too > > internal and global to mention. > > What was the previous memory allocation overhead? On 64-bit builds, it was 16 bytes for AllocSet contexts, 24 bytes for generation contexts and 16 bytes for slab contexts. David
Re: PG 16 draft release notes ready
On Thu, May 25, 2023 at 08:31:29AM +0700, John Naylor wrote: > > On Wed, May 24, 2023 at 8:58 PM Bruce Momjian wrote: > > > > Okay, items split into sections and several merged. I left the > > CPU-specific parts in Source Code, and moved the rest into a merged item > > in General Performance, but moved the JSON item to Data Types. > > It looks like it got moved to Functions actually? > > > > The last one refers to new internal functions, so it could stay in source > code. > > > (Either way, we don't want to imply that arrays of SQL types are > accelerated > > > this way, it's so far only for internal arrays.) > > > > Good point. I called them "C arrays" but it it into the General > > Performance item. > > Looks good to me, although... > > > Allow xid/subxid searches and ASCII string detection to use vector > > operations > (Nathan Bossart) > > Nathan wrote the former, I did the latter. > > Thanks for working on this! Ugh, I have to remember to merge authors when I merge items --- fixed. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: PG 16 draft release notes ready
On Wed, May 24, 2023 at 8:58 PM Bruce Momjian wrote: > > Okay, items split into sections and several merged. I left the > CPU-specific parts in Source Code, and moved the rest into a merged item > in General Performance, but moved the JSON item to Data Types. It looks like it got moved to Functions actually? > > The last one refers to new internal functions, so it could stay in source code. > > (Either way, we don't want to imply that arrays of SQL types are accelerated > > this way, it's so far only for internal arrays.) > > Good point. I called them "C arrays" but it it into the General > Performance item. Looks good to me, although... > Allow xid/subxid searches and ASCII string detection to use vector operations (Nathan Bossart) Nathan wrote the former, I did the latter. Thanks for working on this! -- John Naylor EDB: http://www.enterprisedb.com
Re: PG 16 draft release notes ready
On Wed, May 24, 2023 at 01:43:50PM -0400, Bruce Momjian wrote: > > * Reduce palloc() memory overhead for all memory allocations down to 8 > > bytes on all platforms. (Andres Freund, David Rowley) > > > > This allows more efficient use of memory and is especially useful in > > queries which perform operations (such as sorting or hashing) that > > require more than work_mem. > > Well, this would go in the source code section, but it seems too > internal and global to mention. What was the previous memory allocation overhead? -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: PG 16 draft release notes ready
On Wed, May 24, 2023 at 08:48:56AM +1200, David Rowley wrote: > On Tue, 23 May 2023 at 06:04, Bruce Momjian wrote: > > > > On Mon, May 22, 2023 at 10:59:36AM -0700, Andres Freund wrote: > > > And here it's not just performance, but also memory usage, including > > > steady > > > state memory usage. > > > > Understood. I continue to need help determining which items to include. > > Can you suggest some text? This? > > > > Improve efficiency of memory usage to allow for better scaling > > Maybe something like: > > * Reduce palloc() memory overhead for all memory allocations down to 8 > bytes on all platforms. (Andres Freund, David Rowley) > > This allows more efficient use of memory and is especially useful in > queries which perform operations (such as sorting or hashing) that > require more than work_mem. Well, this would go in the source code section, but it seems too internal and global to mention. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: PG 16 draft release notes ready
On 5/24/23 12:13 AM, David Rowley wrote: On Wed, 24 May 2023 at 15:54, Bruce Momjian wrote: On Wed, May 24, 2023 at 08:37:45AM +1200, David Rowley wrote: On Mon, 22 May 2023 at 07:05, Jonathan S. Katz wrote: * Parallel execution of queries that use `FULL` and `OUTER` joins I think this should be `RIGHT` joins rather than `OUTER` joins. LEFT joins have been parallelizable I think for a long time now. Well, since we can swap left/right easily, why would we not have just have swappted the tables and done the join in the past? I think there are two things missing in my description. First, I need to mention parallel _hash_ join. Second, I think this item is saying that the _inner_ side of a parallel hash join can be an OUTER or FULL join. How about? Allow hash joins to be parallelized where the inner side is processed as an OUTER or FULL join (Melanie Plageman, Thomas Munro) In this case, the inner side is the hashed side. I think Jonathan's text is safe to swap OUTER to RIGHT as it mentions "execution". I made this swap in the release announcement. Thanks! Jonathan OpenPGP_signature Description: OpenPGP digital signature
Re: PG 16 draft release notes ready
On Wed, May 24, 2023 at 04:57:59PM +0200, Erik Rijkers wrote: > Op 5/24/23 om 15:58 schreef Bruce Momjian: > > On Wed, May 24, 2023 at 12:23:02PM +0700, John Naylor wrote: > > > > > > On Wed, May 24, 2023 at 11:19 AM Bruce Momjian wrote: > > Typos: > > 'from standbys servers' should be > 'from standby servers' > > 'reindexedb' should be > 'reindexdb' > (2x: the next line mentions, erroneously, 'reindexedb --system') > > 'created only created' should be > 'only created' > (I think) > > 'could could' should be > 'could' > > 'are now require the role' should be > 'now require the role' > > 'values is' should be > 'value is' > > 'to marked' should be > 'to be marked' All good, patch attached and applied. Updated docs are at: https://momjian.us/pgsql_docs/release-16.html -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you. diff --git a/doc/src/sgml/release-16.sgml b/doc/src/sgml/release-16.sgml index bb92fe5cf9..88d6514ad7 100644 --- a/doc/src/sgml/release-16.sgml +++ b/doc/src/sgml/release-16.sgml @@ -27,7 +27,7 @@ - Allow logical replication from standbys servers + Allow logical replication from standby servers @@ -126,11 +126,11 @@ Author: Michael Paquier -Change REINDEX DATABASE and reindexedb to not process indexes on system catalogs (Simon Riggs) +Change REINDEX DATABASE and reindexdb to not process indexes on system catalogs (Simon Riggs) -Processing such indexes is still possible using REINDEX SYSTEM and reindexedb --system. +Processing such indexes is still possible using REINDEX SYSTEM and reindexdb --system. @@ -593,7 +593,7 @@ Create subscription statistics entries at subscription creation time so stats_re -Previously entries were created only created when the first statistics were reported. +Previously entries were created only when the first statistics were reported. @@ -777,7 +777,7 @@ Simplify permissions for LOCK TABLE (Jeff Davis) -Previously the ability to perform LOCK TABLE at various lock levels was bound to specific query-type permissions. For example, UPDATE could could perform all lock levels except ACCESS SHARE, which +Previously the ability to perform LOCK TABLE at various lock levels was bound to specific query-type permissions. For example, UPDATE could perform all lock levels except ACCESS SHARE, which required SELECT permissions. Now UPDATE can issue all lock levels. MORE? @@ -808,7 +808,7 @@ Restrict the privileges of CREATEROLE roles (Robert Haas) -Previously roles with CREATEROLE privileges could change many aspects of any non-superuser role. Such changes, including adding members, are now require the role requesting the change to have ADMIN OPTION +Previously roles with CREATEROLE privileges could change many aspects of any non-superuser role. Such changes, including adding members, now require the role requesting the change to have ADMIN OPTION permission. @@ -946,7 +946,7 @@ Add dependency tracking of grantors for GRANT records (Robert Haas) -This will guarantee that pg_auth_members.grantor values is always valid. +This guarantees that pg_auth_members.grantor values are always valid. @@ -3017,7 +3017,7 @@ Author: Tom Lane -Allow required extensions to marked as non-relocatable using "no_relocate" (Regina Obe) +Allow required extensions to be marked as non-relocatable using "no_relocate" (Regina Obe)
Re: PG 16 draft release notes ready
Op 5/24/23 om 15:58 schreef Bruce Momjian: On Wed, May 24, 2023 at 12:23:02PM +0700, John Naylor wrote: On Wed, May 24, 2023 at 11:19 AM Bruce Momjian wrote: Typos: 'from standbys servers' should be 'from standby servers' 'reindexedb' should be 'reindexdb' (2x: the next line mentions, erroneously, 'reindexedb --system') 'created only created' should be 'only created' (I think) 'could could' should be 'could' 'are now require the role' should be 'now require the role' 'values is' should be 'value is' 'to marked' should be 'to be marked' thanks, Erik
Re: PG 16 draft release notes ready
On Wed, May 24, 2023 at 12:23:02PM +0700, John Naylor wrote: > > On Wed, May 24, 2023 at 11:19 AM Bruce Momjian wrote: > > > > Second, you might be correct that the section is wrong. I thought of > > CPU instructions as something tied to the compiler, so part of the build > > process or source code, but the point we should be make is that we have > > these acceleration, not how it is implemented. We can move the entire > > group to the "General Performance" section, or we can split it out: > > Splitting out like that seems like a good idea to me. Okay, items split into sections and several merged. I left the CPU-specific parts in Source Code, and moved the rest into a merged item in General Performance, but moved the JSON item to Data Types. Patch attached, and you can see the results at: https://momjian.us/pgsql_docs/release-16.html > The last one refers to new internal functions, so it could stay in source > code. > (Either way, we don't want to imply that arrays of SQL types are accelerated > this way, it's so far only for internal arrays.) Good point. I called them "C arrays" but it it into the General Performance item. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you. commit ad5406246b Author: Bruce Momjian Date: Wed May 24 09:54:34 2023 -0400 doc: PG 16 relnotes, merge and move vector items Reported-by: John Naylor Discussion: https://postgr.es/m/CAFBsxsEPg8L2MmGqavc8JByC=wf_mnkhn-kknfpkcqh0hyd...@mail.gmail.com diff --git a/doc/src/sgml/release-16.sgml b/doc/src/sgml/release-16.sgml index c30c530065..bb92fe5cf9 100644 --- a/doc/src/sgml/release-16.sgml +++ b/doc/src/sgml/release-16.sgml @@ -472,6 +472,28 @@ Author: David Rowley Improve the speed of updating the process title (David Rowley) + + + + + + +Allow xid/subxid searches and ASCII string detection to use vector operations (Nathan Bossart) + + + +ASCII detection is particularly useful for COPY FROM. Vector operations are also used for some C array searches. + + @@ -1781,6 +1803,17 @@ The IS JSON checks include checks for values, arrays, objects, scalars, and uniq + + + + +Allow JSON string parsing to use vector operations (John Naylor) + + + - - - -Allow ASCII string detection to use vector operations (John Naylor) - - - - - - - -Allow JSON string parsing to use vector operations (John Naylor) - - - -ARM? - - - - - - - -Allow array searches to use vector operations (John Naylor) - - - - - - - -Allow xid/subxid searches to use vector operations (Nathan Bossart) - - -
Re: PG 16 draft release notes ready
On Wed, May 24, 2023 at 11:19 AM Bruce Momjian wrote: > > Second, you might be correct that the section is wrong. I thought of > CPU instructions as something tied to the compiler, so part of the build > process or source code, but the point we should be make is that we have > these acceleration, not how it is implemented. We can move the entire > group to the "General Performance" section, or we can split it out: Splitting out like that seems like a good idea to me. > Keep in source code: > > Add support for SSE2 (Streaming SIMD Extensions 2) vector operations on > x86-64 architectures (John Naylor) > > Add support for Advanced SIMD (Single Instruction Multiple Data) (NEON) > instructions on ARM architectures (Nathan Bossart) > > move to General Performance: > > Allow xid/subxid searches to use vector operations (Nathan Bossart) > > Allow ASCII string detection to use vector operations (John Naylor) (The ASCII part is most relevant for COPY FROM, just in case that matters.) > and add these to data types: > > Allow JSON string parsing to use vector operations (John Naylor) > > Allow array searches to use vector operations (John Naylor) The last one refers to new internal functions, so it could stay in source code. (Either way, we don't want to imply that arrays of SQL types are accelerated this way, it's so far only for internal arrays.) -- John Naylor EDB: http://www.enterprisedb.com
Re: PG 16 draft release notes ready
On Tue, May 23, 2023 at 12:14:04PM +0700, John Naylor wrote: > On Tue, May 23, 2023 at 11:26 AM Bruce Momjian wrote: > > > > On Tue, May 23, 2023 at 09:58:30AM +0700, John Naylor wrote: > > > > Allow ASCII string detection to use vector operations on x86-64 > architectures > > > (John Naylor) > > > > Allow JSON string processing to use vector operations on x86-64 > architectures > > > (John Naylor) > > > > > > > > ARM? > > > > > > Arm as well. For anything using 16-byte vectors the two architectures are > > > equivalently supported. For all the applications, I would just say > > > "vector" > or > > > "SIMD". > > > > Okay, I kept "vector". I don't think moving them into performance makes > > sense because there I don't think this would impact user behavior or > > choice, and it can't be controlled. > > Well, these two items were only committed because of measurable speed > increases, and have zero effect on how developers work with "source code", so > that's a category error. > > Whether they rise to the significance of warranting inclusion in release notes > is debatable. Okay, let's dissect this. First, I am excited about these features because I think they show innovation, particularly for high scaling, so I want to highlight this. Second, you might be correct that the section is wrong. I thought of CPU instructions as something tied to the compiler, so part of the build process or source code, but the point we should be make is that we have these acceleration, not how it is implemented. We can move the entire group to the "General Performance" section, or we can split it out: Keep in source code: Add support for SSE2 (Streaming SIMD Extensions 2) vector operations on x86-64 architectures (John Naylor) Add support for Advanced SIMD (Single Instruction Multiple Data) (NEON) instructions on ARM architectures (Nathan Bossart) move to General Performance: Allow xid/subxid searches to use vector operations (Nathan Bossart) Allow ASCII string detection to use vector operations (John Naylor) and add these to data types: Allow JSON string parsing to use vector operations (John Naylor) Allow array searches to use vector operations (John Naylor) -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: PG 16 draft release notes ready
On Wed, 24 May 2023 at 15:54, Bruce Momjian wrote: > > On Wed, May 24, 2023 at 08:37:45AM +1200, David Rowley wrote: > > On Mon, 22 May 2023 at 07:05, Jonathan S. Katz wrote: > > > * Parallel execution of queries that use `FULL` and `OUTER` joins > > > > I think this should be `RIGHT` joins rather than `OUTER` joins. > > > > LEFT joins have been parallelizable I think for a long time now. > > Well, since we can swap left/right easily, why would we not have just > have swappted the tables and done the join in the past? I think there > are two things missing in my description. > > First, I need to mention parallel _hash_ join. Second, I think this > item is saying that the _inner_ side of a parallel hash join can be an > OUTER or FULL join. How about? > > Allow hash joins to be parallelized where the inner side is > processed as an OUTER or FULL join (Melanie Plageman, Thomas Munro) > > In this case, the inner side is the hashed side. I think Jonathan's text is safe to swap OUTER to RIGHT as it mentions "execution". For the release notes, maybe the mention of it can be moved away from "E.1.3.1.1. Optimizer" and put under "E.1.3.1.2. General Performance" and ensure we mention that we're talking about the executor? I'm thinking it might be confusing if we claim that this is something that we switched on in the planner. It was a limitation with the executor which the planner was just onboard with not producing plans for. David
Re: PG 16 draft release notes ready
On Tue, May 23, 2023 at 12:14:04PM +0700, John Naylor wrote: > On Tue, May 23, 2023 at 11:26 AM Bruce Momjian wrote: > > > > Allow xid/subxid searches to use vector operations on x86-64 > architectures > > > (Nathan Bossart) > > > > > > When moved to the performance section, it would be something like "improve > > > scalability when a large number of write transactions are in progress". > > > > Uh, again, see above, this does not impact user behavior or choices. > > So that turns a scalability improvement into "source code"? > > > I assume this is x86-64-only. > > Au contraire, I said "For anything using 16-byte vectors the two architectures > are equivalently supported". It's not clear from looking at individual commit > messages, that's why I piped in to help. Okay, updated text: Allow xid/subxid searches to use vector operations (Nathan Bossart) -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: PG 16 draft release notes ready
On Tue, May 23, 2023 at 06:27:23PM -0400, Jonathan Katz wrote: > On 5/23/23 4:37 PM, David Rowley wrote: > > On Mon, 22 May 2023 at 07:05, Jonathan S. Katz wrote: > > > * Parallel execution of queries that use `FULL` and `OUTER` joins > > > > I think this should be `RIGHT` joins rather than `OUTER` joins. > > > > LEFT joins have been parallelizable I think for a long time now. > > I had grabbed it from this line: > > Allow outer and full joins to be performed in parallel (Melanie Plageman, > Thomas Munro) > > If we want to be specific on RIGHT joins, I can update it in the release > announcement, but it may be too late for the release notes (at least for > beta 1). We will have many more edits before final so I would not worry about adjusting the beta1 wording. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: PG 16 draft release notes ready
On Wed, May 24, 2023 at 08:37:45AM +1200, David Rowley wrote: > On Mon, 22 May 2023 at 07:05, Jonathan S. Katz wrote: > > * Parallel execution of queries that use `FULL` and `OUTER` joins > > I think this should be `RIGHT` joins rather than `OUTER` joins. > > LEFT joins have been parallelizable I think for a long time now. Well, since we can swap left/right easily, why would we not have just have swappted the tables and done the join in the past? I think there are two things missing in my description. First, I need to mention parallel _hash_ join. Second, I think this item is saying that the _inner_ side of a parallel hash join can be an OUTER or FULL join. How about? Allow hash joins to be parallelized where the inner side is processed as an OUTER or FULL join (Melanie Plageman, Thomas Munro) In this case, the inner side is the hashed side. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: PG 16 draft release notes ready
On 5/22/23 4:18 PM, Robert Haas wrote: On Sun, May 21, 2023 at 3:05 PM Jonathan S. Katz wrote: * Support for regular expressions for matching usernames and databases names in `pg_hba.conf`, and user names in `pg_ident.conf` I suggest that this is not a major feature. Perhaps the work that I did to improve CREATEROLE could be considered for inclusion in the major features list. In previous releases, someone with CREATEROLE can hack the PG OS account. Now they can't. In previous releases, someone with CREATEROLE can manage all non-superuser roles, but now they can manage the roles they create (or ones they are given explicit authority to manage). You can even control whether or not such users automatically inherit the privileges of roles they create, as superusers inherit all privileges. There is certainly some argument that this is not a sufficiently significant set of changes to justify a major feature mention, and even if it is, it's not clear to me exactly how it would be best worded. And yet I feel like it's very likely that if we look back on this release in 3 years, those changes will have had a significant impact on many PostgreSQL deployments, above all in the cloud, whereas I think it likely that the ability to have regular expressions in pg_hba.conf and pg_ident.conf will have had very little effect by comparison. Of course, there is always a possibility that I'm over-estimating the impact of my own work. In general, I'm completely fine with people advocating for their own features during this process, in case there's something that I missed. For this case, while I think this work is very impactful, but I don't know if I'd call it a major feature vs. modifying an unintended behavior. Additionally, folks have likely put mitigations in place for this through the years. I'm happy to be convinced otherwise. The regular expressions in the files adds an ability that both we didn't have before, and has been a request I've heard from users with very large deployments. For them, it'll help simplify a lot of their configurations/automations for setting this up en masse. Again, I'm happy to be convinced otherwise. I wanted to use the beta release to allow for us to see 1/ how people ultimately test these things and 2/ help better sift out what will be called a major feature. We could end up shuffling items in the list or completely rewriting it, so it's not set in stone. Thanks, Jonathan OpenPGP_signature Description: OpenPGP digital signature
Re: PG 16 draft release notes ready
On 5/23/23 4:37 PM, David Rowley wrote: On Mon, 22 May 2023 at 07:05, Jonathan S. Katz wrote: * Parallel execution of queries that use `FULL` and `OUTER` joins I think this should be `RIGHT` joins rather than `OUTER` joins. LEFT joins have been parallelizable I think for a long time now. I had grabbed it from this line: Allow outer and full joins to be performed in parallel (Melanie Plageman, Thomas Munro) If we want to be specific on RIGHT joins, I can update it in the release announcement, but it may be too late for the release notes (at least for beta 1). Thanks, Jonathan OpenPGP_signature Description: OpenPGP digital signature
Re: PG 16 draft release notes ready
On Tue, 23 May 2023 at 06:04, Bruce Momjian wrote: > > On Mon, May 22, 2023 at 10:59:36AM -0700, Andres Freund wrote: > > And here it's not just performance, but also memory usage, including steady > > state memory usage. > > Understood. I continue to need help determining which items to include. > Can you suggest some text? This? > > Improve efficiency of memory usage to allow for better scaling Maybe something like: * Reduce palloc() memory overhead for all memory allocations down to 8 bytes on all platforms. (Andres Freund, David Rowley) This allows more efficient use of memory and is especially useful in queries which perform operations (such as sorting or hashing) that require more than work_mem. David
Re: PG 16 draft release notes ready
On Mon, 22 May 2023 at 07:05, Jonathan S. Katz wrote: > * Parallel execution of queries that use `FULL` and `OUTER` joins I think this should be `RIGHT` joins rather than `OUTER` joins. LEFT joins have been parallelizable I think for a long time now. David
Re: PG 16 draft release notes ready
On Tue, May 23, 2023 at 11:26 AM Bruce Momjian wrote: > > On Tue, May 23, 2023 at 09:58:30AM +0700, John Naylor wrote: > > > Allow ASCII string detection to use vector operations on x86-64 architectures > > (John Naylor) > > > Allow JSON string processing to use vector operations on x86-64 architectures > > (John Naylor) > > > > > > ARM? > > > > Arm as well. For anything using 16-byte vectors the two architectures are > > equivalently supported. For all the applications, I would just say "vector" or > > "SIMD". > > Okay, I kept "vector". I don't think moving them into performance makes > sense because there I don't think this would impact user behavior or > choice, and it can't be controlled. Well, these two items were only committed because of measurable speed increases, and have zero effect on how developers work with "source code", so that's a category error. Whether they rise to the significance of warranting inclusion in release notes is debatable. > > > Allow xid/subxid searches to use vector operations on x86-64 architectures > > (Nathan Bossart) > > > > When moved to the performance section, it would be something like "improve > > scalability when a large number of write transactions are in progress". > > Uh, again, see above, this does not impact user behavior or choices. So that turns a scalability improvement into "source code"? > I assume this is x86-64-only. Au contraire, I said "For anything using 16-byte vectors the two architectures are equivalently supported". It's not clear from looking at individual commit messages, that's why I piped in to help. -- John Naylor EDB: http://www.enterprisedb.com
Re: PG 16 draft release notes ready
On Tue, May 23, 2023 at 09:58:30AM +0700, John Naylor wrote: > Hi Bruce, > > > Add support for SSE2 (Streaming SIMD Extensions 2) vector operations on > x86-64 architectures (John Naylor) > > > Add support for Advanced SIMD (Single Instruction Multiple Data) (NEON) > instructions on ARM architectures (Nathan Bossart) > > Nit: It's a bit odd that SIMD is spelled out in only the Arm entry, and > perhaps > expanding the abbreviations can be left out. The issue is that x86-64's SSE2 uses an embedded acronym: SSE2 (Streaming SIMD Extensions 2) so technically it is: SSE2 (Streaming (Single Instruction Multiple Data) Extensions 2 but embedded acronyms is something I wanted to avoid. ;-) > > Allow arrays searches to use vector operations on x86-64 architectures (John > Naylor) > > We can leave out the architecture here (see below). Typo: "array searches" Both fixed. > All the above seem appropriate for the "source code" section, but the > following > entries might be better in the "performance" section: > > > Allow ASCII string detection to use vector operations on x86-64 > > architectures > (John Naylor) > > Allow JSON string processing to use vector operations on x86-64 > > architectures > (John Naylor) > > > > ARM? > > Arm as well. For anything using 16-byte vectors the two architectures are > equivalently supported. For all the applications, I would just say "vector" or > "SIMD". Okay, I kept "vector". I don't think moving them into performance makes sense because there I don't think this would impact user behavior or choice, and it can't be controlled. > And here maybe /processing/parsing/. Done. > > Allow xid/subxid searches to use vector operations on x86-64 architectures > (Nathan Bossart) > > When moved to the performance section, it would be something like "improve > scalability when a large number of write transactions are in progress". Uh, again, see above, this does not impact user behavior or choices. I assume this is x86-64-only. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: PG 16 draft release notes ready
Hi Bruce, > Add support for SSE2 (Streaming SIMD Extensions 2) vector operations on x86-64 architectures (John Naylor) > Add support for Advanced SIMD (Single Instruction Multiple Data) (NEON) instructions on ARM architectures (Nathan Bossart) Nit: It's a bit odd that SIMD is spelled out in only the Arm entry, and perhaps expanding the abbreviations can be left out. > Allow arrays searches to use vector operations on x86-64 architectures (John Naylor) We can leave out the architecture here (see below). Typo: "array searches" All the above seem appropriate for the "source code" section, but the following entries might be better in the "performance" section: > Allow ASCII string detection to use vector operations on x86-64 architectures (John Naylor) > Allow JSON string processing to use vector operations on x86-64 architectures (John Naylor) > > ARM? Arm as well. For anything using 16-byte vectors the two architectures are equivalently supported. For all the applications, I would just say "vector" or "SIMD". And here maybe /processing/parsing/. > Allow xid/subxid searches to use vector operations on x86-64 architectures (Nathan Bossart) When moved to the performance section, it would be something like "improve scalability when a large number of write transactions are in progress". -- John Naylor EDB: http://www.enterprisedb.com
Re: PG 16 draft release notes ready
On Sun, May 21, 2023 at 3:05 PM Jonathan S. Katz wrote: > * Support for regular expressions for matching usernames and databases > names in `pg_hba.conf`, and user names in `pg_ident.conf` I suggest that this is not a major feature. Perhaps the work that I did to improve CREATEROLE could be considered for inclusion in the major features list. In previous releases, someone with CREATEROLE can hack the PG OS account. Now they can't. In previous releases, someone with CREATEROLE can manage all non-superuser roles, but now they can manage the roles they create (or ones they are given explicit authority to manage). You can even control whether or not such users automatically inherit the privileges of roles they create, as superusers inherit all privileges. There is certainly some argument that this is not a sufficiently significant set of changes to justify a major feature mention, and even if it is, it's not clear to me exactly how it would be best worded. And yet I feel like it's very likely that if we look back on this release in 3 years, those changes will have had a significant impact on many PostgreSQL deployments, above all in the cloud, whereas I think it likely that the ability to have regular expressions in pg_hba.conf and pg_ident.conf will have had very little effect by comparison. Of course, there is always a possibility that I'm over-estimating the impact of my own work. -- Robert Haas EDB: http://www.enterprisedb.com
Re: PG 16 draft release notes ready
On Mon, May 22, 2023 at 11:03:31AM -0700, Andres Freund wrote: > On 2023-05-21 22:52:09 -0400, Bruce Momjian wrote: > > Do average users know heap and index files are both relations? That > > seems too abstract so I spelled out heap and index pages. > > I don't know about average users - but I think users that read the release > notes do know. > > I am a bit on the fence about "addition" vs "extending" - for me it's not > clear what "adding pages" really means, but I might be too deep into this. I am worried "extending" and "extensions" might be too close a wording since we often mention extensions. I tried "increase the file eize" but that seemed wordy. Ideas? Personally, while I consider heap and indexes to be both relations at the SQL level, at the file system level I tend to think of them as different, but perhaps that is just me. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: PG 16 draft release notes ready
On Mon, May 22, 2023 at 10:59:36AM -0700, Andres Freund wrote: > On 2023-05-21 22:46:58 -0400, Bruce Momjian wrote: > > > For the above two items, I mention items that would change user > > > like new features or changes that are significant enough that they would > > > change user behavior. For example, if a new join method increases > > > performance by 5x, that could change user behavior. Based on the quoted > > > numbers above, I didn't think "hash now faster" would be appropriate to > > > mention. Right? > > I continue, as in past releases, to think that this is a bad policy. For > existing workloads performance improvements are commonly a more convincing > reason to upgrade than new features - they allow users to scale the workload > further, without needing application changes. > > Of course there are performance improvement that are too miniscule to be worth > mentioning, but it's not a common case. > > And here it's not just performance, but also memory usage, including steady > state memory usage. Understood. I continue to need help determining which items to include. Can you suggest some text? This? Improve efficiency of memory usage to allow for better scaling -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: PG 16 draft release notes ready
Hi, On 2023-05-21 22:52:09 -0400, Bruce Momjian wrote: > On Sun, May 21, 2023 at 10:13:41AM -0700, Andres Freund wrote: > > Hi, > > > > Thanks for the release notes! > > > > > > > > > > > > > > > > > Allow more efficient addition of multiple heap and index pages (Andres > > > Freund) > > > > > > > > > > While the case of extending by multiple pages improved the most, even > > extending by a single page at a time got a good bit more scalable. Maybe > > just > > "Improve efficiency of extending relations"? > > Do average users know heap and index files are both relations? That > seems too abstract so I spelled out heap and index pages. I don't know about average users - but I think users that read the release notes do know. I am a bit on the fence about "addition" vs "extending" - for me it's not clear what "adding pages" really means, but I might be too deep into this. Greetings, Andres Freund
Re: PG 16 draft release notes ready
I have added the major features to the release notes with the attached patch. --- On Sun, May 21, 2023 at 07:53:38PM -0400, Jonathan Katz wrote: > On 5/21/23 3:04 PM, Jonathan S. Katz wrote: > > On 5/18/23 4:49 PM, Bruce Momjian wrote: > > > I have completed the first draft of the PG 16 release notes. > > > > One thing that we could attempt for this beta is to include a > > prospective list of "major features + enhancements." Of course it can > > change before the GA, but it'll give readers some idea of things to > > test. > > > > I'd propose the following (in no particular order): > > > > * General performance improvements for read-heavy workloads (looking for > > clarification for that in[1]) > > Per [1] this sounds like it should be: > > * Optimization to reduce overall memory usage, including general performance > improvements. > > We can get more specific for the GA. > > Thanks, > > Jonathan > > [1] > https://www.postgresql.org/message-id/5749E807-A5B7-4CC7-8282-84F6F0D4D1D0%40anarazel.de > -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you. commit 60751aa503 Author: Bruce Momjian Date: Mon May 22 13:58:24 2023 -0400 doc: PG 16 relnotes, add major features list Reported-by: Jonathan Katz Discussion: https://postgr.es/m/2fd2cc0e-df39-3e77-8fcf-35aad5796...@postgresql.org diff --git a/doc/src/sgml/release-16.sgml b/doc/src/sgml/release-16.sgml index 893cd8ddb0..3d96bd6e6d 100644 --- a/doc/src/sgml/release-16.sgml +++ b/doc/src/sgml/release-16.sgml @@ -18,7 +18,49 @@ -ADD HERE + + + + Allow parallel execution of queries with OUTER and FULL joins + + + + + + Allow logical replication from standbys servers + + + + + + Allow logical replication subscribers to apply large transactions in parallel + + + + + + Allow monitoring of I/O statistics using the new pg_stat_io view + + + + + + Add SQL/JSON constructors and identity functions + + + + + + Improve performance of vacuum freezing + + + + + + Add support for regular expression matching of user and database names in pg_hba.conf, and user names in pg_ident.conf + + + @@ -280,7 +322,7 @@ Author: Thomas Munro -Allow full and outer joins to be performed in parallel (Melanie Plageman, Thomas Munro) +Allow outer and full joins to be performed in parallel (Melanie Plageman, Thomas Munro)
Re: PG 16 draft release notes ready
Hi, On 2023-05-21 22:46:58 -0400, Bruce Momjian wrote: > > Looking through the release notes, I didn't see an entry for > > > > commit c6e0fe1f2a08505544c410f613839664eea9eb21 > > Author: David Rowley > > Date: 2022-08-29 17:15:00 +1200 > > > > Improve performance of and reduce overheads of memory management > > > > even though I think that's one of the more impactful improvements. What was > > the reason for leaving that out? > > If you read my previous email: > > > For the above two items, I mention items that would change user > > like new features or changes that are significant enough that they would > > change user behavior. For example, if a new join method increases > > performance by 5x, that could change user behavior. Based on the quoted > > numbers above, I didn't think "hash now faster" would be appropriate to > > mention. Right? I continue, as in past releases, to think that this is a bad policy. For existing workloads performance improvements are commonly a more convincing reason to upgrade than new features - they allow users to scale the workload further, without needing application changes. Of course there are performance improvement that are too miniscule to be worth mentioning, but it's not a common case. And here it's not just performance, but also memory usage, including steady state memory usage. > I can see this item as a big win, but I don't know how to describe it in a way > that is helpful for the user to know. In doubt the subject of the commit would just work IMO. Greetings, Andres Freund
Re: PG 16 draft release notes ready
On Sun, May 21, 2023 at 10:13:41AM -0700, Andres Freund wrote: > Hi, > > Thanks for the release notes! > > > > > > > > > > > Allow more efficient addition of multiple heap and index pages (Andres > > Freund) > > > > > > While the case of extending by multiple pages improved the most, even > extending by a single page at a time got a good bit more scalable. Maybe just > "Improve efficiency of extending relations"? Do average users know heap and index files are both relations? That seems too abstract so I spelled out heap and index pages. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: PG 16 draft release notes ready
On Mon, May 22, 2023 at 09:03:11AM +0800, jian he wrote: > In E.1.2. Migration to Version 16, probably need mention, some > privilege command cannot restore. > if new cluster bootstrap superuser name is not the same as old one. "GRANT x > TO > y GRANTED BY no_bootstrap_superuser; " will have error. > > ---pg15 dump content. > CREATE ROLE jian; > ALTER ROLE jian WITH SUPERUSER INHERIT CREATEROLE CREATEDB LOGIN REPLICATION > BYPASSRLS; > CREATE ROLE regress_priv_user1; > ALTER ROLE regress_priv_user1 WITH NOSUPERUSER INHERIT NOCREATEROLE NOCREATEDB > LOGIN NOREPLICATION NOBYPASSRLS; > CREATE ROLE regress_priv_user2; > ALTER ROLE regress_priv_user2 WITH NOSUPERUSER INHERIT NOCREATEROLE NOCREATEDB > LOGIN NOREPLICATION NOBYPASSRLS; > CREATE ROLE su1; > ALTER ROLE su1 WITH SUPERUSER INHERIT CREATEROLE NOCREATEDB LOGIN > NOREPLICATION > NOBYPASSRLS; > GRANT regress_priv_user1 TO regress_priv_user2 GRANTED BY su1; > > ---restore in pg16 > \i /home/jian/Desktop/dumpall_schema.sql > 2023-05-22 08:46:00.170 CST [456584] ERROR: permission denied to grant > privileges as role "su1" > 2023-05-22 08:46:00.170 CST [456584] DETAIL: The grantor must have the ADMIN > option on role "regress_priv_user1". > 2023-05-22 08:46:00.170 CST [456584] STATEMENT: GRANT regress_priv_user1 TO > regress_priv_user2 GRANTED BY su1; > psql:/home/jian/Desktop/dumpall_schema.sql:32: ERROR: permission denied to > grant privileges as role "su1" > DETAIL: The grantor must have the ADMIN option on role "regress_priv_user1". Agreed, new text: Prevent removal of superuser privileges for the bootstrap user (Robert Haas) --> Restoring such users could lead to errors. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: PG 16 draft release notes ready
On Sun, May 21, 2023 at 10:13:41AM -0700, Andres Freund wrote: > Hi, > > Thanks for the release notes! > > > > > > > > > > > Allow more efficient addition of multiple heap and index pages (Andres > > Freund) > > > > > > While the case of extending by multiple pages improved the most, even > extending by a single page at a time got a good bit more scalable. Maybe just > "Improve efficiency of extending relations"? Okay, I made this change: -Allow more efficient addition of multiple heap and index pages (Andres Freund) +Allow more efficient addition of heap and index pages (Andres Freund) > I think: > > > > > > > > > > > Allow logical decoding on standbys (Bertrand Drouvot, Andres Freund, Amit > > Khandekar) > > > > > > pretty much includes: > > > > > > > > > > > Allow invalidation of replication slots due to row removal, wal_level, and > > conflicts (Bertrand Drouvot, Andres Freund, Amit Khandekar) > > > > as it is a prerequisite. Okay, I merged the commit entries and the authors, and removed the item. > I'd probably also merge > > > > > > > > > > > Add function pg_log_standby_snapshot() to force creation of a WAL snapshot > > (Bertrand Drouvot) > > > > > > > > WAL snapshots are required for logical slot creation so this function > > speeds their creation on standbys. > > > > > > As there really isn't a use case outside of logical decoding on a standby. Okay new merged item is: Allow logical decoding on standbys (Bertrand Drouvot, Andres Freund, Amit Khandekar, Bertrand Drouvot) New function pg_log_standby_snapshot() forces creation of WAL snapshots. Snapshots are required for logical slot creation so this function speeds their creation on standbys. > > > > > > > > > > Prevent extension libraries from export their symbols by default (Andres > > Freund, Tom Lane) > > > > > > s/export/exporting/? Seems Tom's patch already fixed that. > Looking through the release notes, I didn't see an entry for > > commit c6e0fe1f2a08505544c410f613839664eea9eb21 > Author: David Rowley > Date: 2022-08-29 17:15:00 +1200 > > Improve performance of and reduce overheads of memory management > > even though I think that's one of the more impactful improvements. What was > the reason for leaving that out? If you read my previous email: > For the above two items, I mention items that would change user > like new features or changes that are significant enough that they would > change user behavior. For example, if a new join method increases > performance by 5x, that could change user behavior. Based on the quoted > numbers above, I didn't think "hash now faster" would be appropriate to > mention. Right? I can see this item as a big win, but I don't know how to describe it in a way that is helpful for the user to know. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: PG 16 draft release notes ready
In E.1.2. Migration to Version 16, probably need mention, some privilege command cannot restore. if new cluster bootstrap superuser name is not the same as old one. "GRANT x TO y GRANTED BY no_bootstrap_superuser; " will have error. ---pg15 dump content. CREATE ROLE jian; ALTER ROLE jian WITH SUPERUSER INHERIT CREATEROLE CREATEDB LOGIN REPLICATION BYPASSRLS; CREATE ROLE regress_priv_user1; ALTER ROLE regress_priv_user1 WITH NOSUPERUSER INHERIT NOCREATEROLE NOCREATEDB LOGIN NOREPLICATION NOBYPASSRLS; CREATE ROLE regress_priv_user2; ALTER ROLE regress_priv_user2 WITH NOSUPERUSER INHERIT NOCREATEROLE NOCREATEDB LOGIN NOREPLICATION NOBYPASSRLS; CREATE ROLE su1; ALTER ROLE su1 WITH SUPERUSER INHERIT CREATEROLE NOCREATEDB LOGIN NOREPLICATION NOBYPASSRLS; GRANT regress_priv_user1 TO regress_priv_user2 GRANTED BY su1; ---restore in pg16 \i /home/jian/Desktop/dumpall_schema.sql 2023-05-22 08:46:00.170 CST [456584] ERROR: permission denied to grant privileges as role "su1" 2023-05-22 08:46:00.170 CST [456584] DETAIL: The grantor must have the ADMIN option on role "regress_priv_user1". 2023-05-22 08:46:00.170 CST [456584] STATEMENT: GRANT regress_priv_user1 TO regress_priv_user2 GRANTED BY su1; psql:/home/jian/Desktop/dumpall_schema.sql:32: ERROR: permission denied to grant privileges as role "su1" DETAIL: The grantor must have the ADMIN option on role "regress_priv_user1".
Re: PG 16 draft release notes ready
On 5/21/23 3:04 PM, Jonathan S. Katz wrote: On 5/18/23 4:49 PM, Bruce Momjian wrote: I have completed the first draft of the PG 16 release notes. One thing that we could attempt for this beta is to include a prospective list of "major features + enhancements." Of course it can change before the GA, but it'll give readers some idea of things to test. I'd propose the following (in no particular order): * General performance improvements for read-heavy workloads (looking for clarification for that in[1]) Per [1] this sounds like it should be: * Optimization to reduce overall memory usage, including general performance improvements. We can get more specific for the GA. Thanks, Jonathan [1] https://www.postgresql.org/message-id/5749E807-A5B7-4CC7-8282-84F6F0D4D1D0%40anarazel.de OpenPGP_signature Description: OpenPGP digital signature
Re: PG 16 draft release notes ready
On 5/21/23 3:24 PM, Andres Freund wrote: Hi, On May 21, 2023 11:46:56 AM PDT, "Jonathan S. Katz" wrote: On 5/21/23 1:13 PM, Andres Freund wrote: Looking through the release notes, I didn't see an entry for commit c6e0fe1f2a08505544c410f613839664eea9eb21 Author: David Rowley Date: 2022-08-29 17:15:00 +1200 Improve performance of and reduce overheads of memory management even though I think that's one of the more impactful improvements. What was the reason for leaving that out? IIUC in[1], would this "just speed up" read-heavy workloads? I don't think so. It can speed up write workloads as well. But more importantly it can noticeably reduce memory usage, including for things like the relcache. Cool! I'll dive more into the thread later to learn more. Jonathan OpenPGP_signature Description: OpenPGP digital signature
Re: PG 16 draft release notes ready
On Sun, May 21, 2023 at 01:11:05PM -0400, Tom Lane wrote: > Bruce Momjian writes: > > On Sun, May 21, 2023 at 09:30:01PM +0900, Ian Lawrence Barwick wrote: > >> 2ceea5adb Accept "+infinity" in date and timestamp[tz] input. > > > I have this but didn't add that commit, added. > > That's really not related to the commit you added it to... > > I don't have time today to read through all the relnotes, but I went > through those that have my name on them. Suggested wording modifications > attached. These were all good, patch applied, thanks. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: PG 16 draft release notes ready
Hi, On May 21, 2023 11:46:56 AM PDT, "Jonathan S. Katz" wrote: >On 5/21/23 1:13 PM, Andres Freund wrote: > >> >> Looking through the release notes, I didn't see an entry for >> >> commit c6e0fe1f2a08505544c410f613839664eea9eb21 >> Author: David Rowley >> Date: 2022-08-29 17:15:00 +1200 >>Improve performance of and reduce overheads of memory management >> >> even though I think that's one of the more impactful improvements. What was >> the reason for leaving that out? > >IIUC in[1], would this "just speed up" read-heavy workloads? I don't think so. It can speed up write workloads as well. But more importantly it can noticeably reduce memory usage, including for things like the relcache. Andres -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
Re: PG 16 draft release notes ready
On 5/18/23 4:49 PM, Bruce Momjian wrote: I have completed the first draft of the PG 16 release notes. One thing that we could attempt for this beta is to include a prospective list of "major features + enhancements." Of course it can change before the GA, but it'll give readers some idea of things to test. I'd propose the following (in no particular order): * General performance improvements for read-heavy workloads (looking for clarification for that in[1]) * Parallel execution of queries that use `FULL` and `OUTER` joins * Logical replication allowed from read-only standbys * Logical replication subscribers can apply large transactions in parallel * Monitoring of I/O statistics through the `pg_stat_io` view * Addition of SQL/JSON constructors and identity functions * Optimizations to the vacuum freezing strategy * Support for regular expressions for matching usernames and databases names in `pg_hba.conf`, and user names in `pg_ident.conf` The above is tossing items at the wall, and I'm OK with any or all being modified or replaced. Thanks, Jonathan [1] https://postgr.es/m/CAApHDvpjauCRXcgcaL6+e3eqecEHoeRm9D-kcbuvBitgPnW=v...@mail.gmail.com OpenPGP_signature Description: OpenPGP digital signature
