Re: Unknown annotation '-cim' in source code
On Thu, Jul 4, 2024 at 04:18:43PM +0200, Daniel Gustafsson wrote: > > Searching for "cimarron postgres" returns > > https://en.wikipedia.org/wiki/Illustra, which mentions a Cimarron Taylor > > as one of Stonebraker's students, but I can't find anything else > > relevant in a few minutes of searching. > > That seems to match up. There is a Cimarron Taylor on LinkedIN who was > "Programmer/Analyst" at U.C. Berkeley Database Research Group in January 1978 > through January 1990. And the Stonebraker video lists Cimarron Taylor as one of the 39 Berkely contributors: https://momjian.us/main/blogs/pgblog/2015.html#August_5_2015 https://www.youtube.com/watch?v=BbGeKi6T6QI=4269s -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: Changing the state of data checksums in a running cluster
On Wed, Jul 3, 2024 at 01:20:10PM +0200, Tomas Vondra wrote: > > * Restartability - the initial version of the patch did not support stateful > > restarts, a shutdown performed (or crash) before checksums were enabled > > would > > result in a need to start over from the beginning. This was deemed the safe > > orchestration method. The lack of this feature was seen as serious > > drawback, > > so it was added. Subsequent review instead found the patch to be too > > complicated with a too large featureset. I thihk there is merit to both of > > these arguments: being able to restart is a great feature; and being able to > > reason about the correctness of a smaller patch is also great. As of this > > submission I have removed the ability to restart to keep the scope of the > > patch > > small (which is where the previous version was, which received no review > > after > > the removal). The way I prefer to frame this is to first add scaffolding > > and > > infrastructure (this patch) and leave refinements and add-on features > > (restartability, but also others like parallel workers, optimizing rare > > cases, > > etc) for follow-up patches. > > > > I 100% support this approach. Yes, I was very disappointed when restartability sunk the patch, and I saw this as another case where saying "yes" to every feature improvement can lead to failure. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: Should we document how column DEFAULT expressions work?
On Fri, Jul 5, 2024 at 05:03:35PM -0400, Tom Lane wrote:
> Bruce Momjian writes:
> > Well, 'now()' certainly _looks_ like a function call, though it isn't.
> > The fact that 'now()'::timestamptz and 'now'::timestamptz generate
> > volatile results via a function call was my point.
>
> The only reason 'now()'::timestamptz works is that timestamptz_in
> ignores irrelevant punctuation (or what it thinks is irrelevant,
> anyway). I do not think we should include examples that look like
> that, because it will further confuse readers who don't already
> have a solid grasp of how this works.
Wow, I see that now:
test=> SELECT 'now('::timestamptz;
timestamptz
---
2024-07-05 17:04:33.457915-04
If I remove the 'now()' mention in the docs, patch attached, I am
concerned people will be confused whether it is the removal of the
single quotes or the use of "()" which causes insert-time evaluation,
and they might try 'now()'.
--
Bruce Momjian https://momjian.us
EDB https://enterprisedb.com
Only you can decide what is important to you.
diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml
index f19306e7760..4d47248fccf 100644
--- a/doc/src/sgml/ref/create_table.sgml
+++ b/doc/src/sgml/ref/create_table.sgml
@@ -888,6 +888,13 @@ WITH ( MODULUS numeric_literal, REM
match the data type of the column.
+
+ Note, a string that returns a volatile result once cast to a data
+ type, like 'now'::timestamptz, is evaluated at
+ table creation time, while now()::timestamptz
+ (without quotes) is evaluated at data insertion time.
+
+
The default expression will be used in any insert operation that
does not specify a value for the column. If there is no default
Re: Should we document how column DEFAULT expressions work?
On Fri, Jul 5, 2024 at 04:50:32PM -0400, Tom Lane wrote: > Bruce Momjian writes: > > Also interestingly, "now" without quotes requires parentheses to make it > > a function call: > > I'm not sure why you find that surprising, or why you think that > 'now()'::timestamptz is a function call. (Well, it is a call of > timestamptz_in, but not of the SQL function now().) Documentation > that is equally confused won't help any. Well, 'now()' certainly _looks_ like a function call, though it isn't. The fact that 'now()'::timestamptz and 'now'::timestamptz generate volatile results via a function call was my point. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Fri, Jul 5, 2024 at 07:51:38PM +0200, Matthias van de Meent wrote: > Hi, > > I noticed that PG17's release note for commit cafe10565 is "Allow psql > connections to be canceled with control-C (Tristan Partin)", but this > summary seems wrong to me. > > We already had ^C connection (query) cancellation for quite some time > before this patch. What's new with that patch, is that we now also can > cancel connection attempts with ^C while we're still connecting (i.e., > we haven't yet authenticated and are trying to move the connection > state forward). > I think a better wording would be "Allow psql connection attempts to > be canceled with control-C (Tristan Partin)", or "Allow psql > connections to be canceled with control-C while psql is still > connecting (Tristan Partin)". I see your point. I committed a change to use this wording: Allow psql connection attempts to be canceled with control-C (Tristan Partin) -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: Should we document how column DEFAULT expressions work?
On Mon, Jul 1, 2024 at 02:52:42PM +1200, David Rowley wrote: > On Mon, 1 Jul 2024 at 13:41, David G. Johnston > wrote: > > I presume the relatively new atomic SQL functions pose a similar hazard. > > Do you have an example of this? > > > The fact that 'now()'::timestamp fails to fail doesn't help... > > If that's the case, maybe a tiny step towards what Peter proposed is > just to make trailing punctuation fail for timestamp special values in > v18. I dug into this and I have a suggestion at the end. First, the special values like 'now' are the only values that can be optionally quoted: SELECT current_timestamp::timestamptz; current_timestamp --- 2024-07-05 15:15:22.692072-04 SELECT 'current_timestamp'::timestamptz; ERROR: invalid input syntax for type timestamp with time zone: "current_timestamp" Also interestingly, "now" without quotes requires parentheses to make it a function call: SELECT 'now'::timestamptz; timestamptz --- 2024-07-05 15:17:11.394182-04 SELECT 'now()'::timestamptz; timestamptz --- 2024-07-05 15:17:15.201621-04 SELECT now()::timestamptz; now --- 2024-07-05 15:17:21.925611-04 SELECT now::timestamptz; ERROR: column "now" does not exist LINE 1: SELECT now::timestamptz; ^ And the quoting shows "now" evaluation at function creation time: CREATE OR REPLACE FUNCTION testnow() RETURNS timestamptz LANGUAGE SQL RETURN 'now'::timestamptz; SELECT testnow(); SELECT pg_sleep(5); SELECT testnow(); testnow --- 2024-07-05 15:19:38.915255-04 testnow --- 2024-07-05 15:19:38.915255-04 -- same --- CREATE OR REPLACE FUNCTION testnow() RETURNS timestamptz LANGUAGE SQL RETURN 'now()'::timestamptz; SELECT testnow(); SELECT pg_sleep(5); SELECT testnow(); testnow --- 2024-07-05 15:20:41.475997-04 testnow --- 2024-07-05 15:20:41.475997-04 -- same --- CREATE OR REPLACE FUNCTION testnow() RETURNS timestamptz LANGUAGE SQL RETURN now()::timestamptz; SELECT testnow(); SELECT pg_sleep(5); SELECT testnow(); testnow --- 2024-07-05 15:21:18.204574-04 testnow --- 2024-07-05 15:21:23.210442-04 -- different I don't think we can bounce people around to different sections to explain this --- I think we need text in the CREATE TABLE ... DEFAULT section. I think the now() case is unusual since there are few cases where function calls can be put inside of single quotes. I have written the attached patch to clarify the behavior. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you. diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index f19306e7760..9bab4ec141e 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -888,6 +888,14 @@ WITH ( MODULUS numeric_literal, REM match the data type of the column. + + Note, a string that returns a volatile result once cast to + a data type, like 'now'::timestamptz and + 'now()'::timestamptz, is evaluated at table + creation time, while now()::timestamptz (without + quotes) is evaluated at data insertion time. + + The default expression will be used in any insert operation that does not specify a value for the column. If there is no default
Re: Partial aggregates pushdown
On Sun, Jun 30, 2024 at 09:42:19PM +, fujii.y...@df.mitsubishielectric.co.jp wrote: > On Mon, Jun 24, 2024 at 6:09?PM Jelte Fennema-Nio wrote: > > 4. Related to 3, I think it would be good to have some tests of > > PARTIAL_AGGREGATE that don't involve postgres_fdw at all. I also > > spotted some comments too that mention FDW, even though they apply to > > the "pure" PARTIAL_AGGREGATE code. > > 5. This comment now seems incorrect: > > -* Apply the agg's finalfn if one is provided, else return transValue. > > +* If the agg's finalfn is provided and PARTIAL_AGGREGATE keyword is > > +* not specified, apply the agg's finalfn. > > +* If PARTIAL_AGGREGATE keyword is specified and the transValue type > > +* is internal, apply the agg's serialfn. In this case the agg's > > +* serialfn must not be invalid. Otherwise return transValue. > > > > 6. These errors are not on purpose afaict (if they are a comment in > > the test would be good to explain why) > > > > +SELECT b, avg(a), max(a), count(*) FROM pagg_tab GROUP BY b ORDER BY 1; > > +ERROR: could not connect to server "loopback" > > +DETAIL: invalid connection option "partial_aggregate_support" > Fixed. Is there a reason the documentation is no longer a part of this patch? Can I help you keep it current? -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: psql (PostgreSQL) 17beta2 (Debian 17~beta2-1.pgdg+~20240625.1534.g23c5a0e) Failed to retrieve data from the server..
On Wed, Jun 26, 2024 at 09:59:35AM -0400, Bruce Momjian wrote: > On Wed, Jun 26, 2024 at 12:06:13AM -0400, Tom Lane wrote: > > Bruce Momjian writes: > > > On Tue, Jun 25, 2024 at 05:52:47PM -0700, David G. Johnston wrote: > > >> We seem to be missing a release note item for the catalog breakage done > > >> in > > >> f696c0c > > >> https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=f696c0cd5f > > > > > It seemed too internal to mention in the release notes --- more of an > > > infrastructure change, but I can add it if I was wrong about this. > > > > As this breakage demonstrates, that change is quite > > application-visible. It needs an entry under incompatibilities. > > Okay, will do today. Done. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: psql (PostgreSQL) 17beta2 (Debian 17~beta2-1.pgdg+~20240625.1534.g23c5a0e) Failed to retrieve data from the server..
On Wed, Jun 26, 2024 at 12:06:13AM -0400, Tom Lane wrote: > Bruce Momjian writes: > > On Tue, Jun 25, 2024 at 05:52:47PM -0700, David G. Johnston wrote: > >> We seem to be missing a release note item for the catalog breakage done in > >> f696c0c > >> https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=f696c0cd5f > > > It seemed too internal to mention in the release notes --- more of an > > infrastructure change, but I can add it if I was wrong about this. > > As this breakage demonstrates, that change is quite > application-visible. It needs an entry under incompatibilities. Okay, will do today. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: psql (PostgreSQL) 17beta2 (Debian 17~beta2-1.pgdg+~20240625.1534.g23c5a0e) Failed to retrieve data from the server..
On Tue, Jun 25, 2024 at 05:52:47PM -0700, David G. Johnston wrote: > On Tue, Jun 25, 2024 at 5:36 PM André Verwijs wrote: > > psql (PostgreSQL) 17beta2 (Debian 17~beta2-1.pgdg+~20240625.1534.g23c5a0e) > > > column "daticulocale" does not exist > LINE 5: datconnlimit, daticulocale, daticurules, datcollversion, > ^ > HINT: Perhaps you meant to reference the column "db.datlocale". > -- > > > INFO PGADMIN: > > > > We seem to be missing a release note item for the catalog breakage done in > f696c0c > > https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=f696c0cd5f It seemed too internal to mention in the release notes --- more of an infrastructure change, but I can add it if I was wrong about this. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: Proposal for Updating CRC32C with AVX-512 Algorithm.
On Tue, Jun 25, 2024 at 05:41:12PM +, Amonson, Paul D wrote: > > It would be good to know exactly what, if any, changes the Intel > > lawyers want us to make to our license if we accept this patch. > > I asked about this and there is nothing Intel requires here license > wise. They believe that there is nothing wrong with including Clause-3 > BSD like licenses under the PostgreSQL license. They only specified > that for the source file, the applying license need to be present > either as a link (which was previously discouraged in this thread) > or the full text. Please note that I checked and for this specific > Chromium license there is not SPDX codename so the entire text is > required. Okay, that is very interesting. Yes, we will have no problem reproducing the exact license text in the source code. I think we can remove the license issue as a blocker for this patch. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: New standby_slot_names GUC in PG 17
On Sat, Jun 22, 2024 at 03:17:03PM +0530, Amit Kapila wrote: > On Sat, Jun 22, 2024 at 1:49 AM Nathan Bossart > wrote: > > > > On Fri, Jun 21, 2024 at 03:50:00PM -0400, Tom Lane wrote: > > >>>>> Allow specification of physical standbys that must be synchronized > > >>>>> before they are visible to subscribers (Hou Zhijie, Shveta Malik) > > > > > > it seems like the name ought to have some connection to > > > synchronization. Perhaps something like "synchronized_standby_slots"? > > > > IMHO that might be a bit too close to synchronous_standby_names. > > > > Right, but better than the current one. The other possibility could be > wait_for_standby_slots. FYI, changing this GUC name could force an initdb because postgresql.conf would have the old name and removing the comment to change it would cause an error. Therefore, we should change it ASAP. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
New standby_slot_names GUC in PG 17
The release notes have this item: Allow specification of physical standbys that must be synchronized before they are visible to subscribers (Hou Zhijie, Shveta Malik) The new server variable is standby_slot_names. Is standby_slot_names an accurate name for this GUC? It seems too generic. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
PG 17 and GUC variables
FYI, looking at the release notes, I see 15 GUC variables added in this release, and two removed. That 15 number seemed unusually high so I thought I would report it. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: Proposal for Updating CRC32C with AVX-512 Algorithm.
On Tue, Jun 18, 2024 at 02:00:34PM -0400, Bruce Momjian wrote: > While the license we are concerned about does not have this clause, it > does have: > >2. Redistributions in binary form must reproduce the above > copyright notice, this list of conditions and the following > disclaimer in the documentation and/or other materials provided > with the distribution. > > I assume that must also include the name of the copyright holder. > > I think that means we need to mention The Regents of the University of > California in our copyright notice, which we do. However several > non-Regents of the University of California copyright holder licenses > exist in our source tree, and accepting this AVX-512 patch would add > another one. Specifically, I see existing entries for: > > Aaron D. Gifford > Board of Trustees of the University of Illinois > David Burren > Eric P. Allman > Jens Schweikhardt > Marko Kreen > Sun Microsystems, Inc. > WIDE Project > > Now, some of these are these names plus Berkeley, and some are just the > names above. In summary, either we are doing something wrong in how we list copyrights in our documentation, or we don't need to make any changes for this Intel patch. Our license is at: https://www.postgresql.org/about/licence/ The Intel copyright in the source code is: * Copyright 2017 The Chromium Authors * Copyright (c) 2024, Intel(r) Corporation * * Use of this source code is governed by a BSD-style license that can be * found in the Chromium source repository LICENSE file. * https://chromium.googlesource.com/chromium/src/+/refs/heads/main/LICENSE and the URL contents are: // Copyright 2015 The Chromium Authors // // Redistribution and use in source and binary forms, with or without // modification, are permitted provided that the following conditions are // met: // //* Redistributions of source code must retain the above copyright // notice, this list of conditions and the following disclaimer. //* Redistributions in binary form must reproduce the above // copyright notice, this list of conditions and the following disclaimer // in the documentation and/or other materials provided with the // distribution. //* Neither the name of Google LLC nor the names of its // contributors may be used to endorse or promote products derived from // this software without specific prior written permission. // // THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS // "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT // LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR // A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT // OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, // SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT // LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, // DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY // THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT // (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE // OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. Google LLC is added to clause three, and I assume Intel is also covered by this because it is considered "the names of its contributors", maybe? It would be good to know exactly what, if any, changes the Intel lawyers want us to make to our license if we accept this patch. There are also different versions of clause three in our source tree. The Postgres license only lists the University of California in our equivalent of clause three, meaning that there are three-clause BSD licenses in our source tree that reference entities that we don't reference in the Postgres license. Oddly, the Postgres license doesn't even disclaim warranties for the PostgreSQL Global Development Group, only for Berkeley. An even bigger issue is that we are distributing 3-clause BSD licensed software under the Postgres license, which is not the 3-clause BSD license. I think we were functioning under the assuption that the licenses are compatibile, so can be combined, which is true, but I don't think we can assume the individual licenses can be covered by our one license, can we? -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: Proposal for Updating CRC32C with AVX-512 Algorithm.
On Tue, Jun 18, 2024 at 01:20:50PM -0400, Bruce Momjian wrote:
> On Tue, Jun 18, 2024 at 05:14:08PM +, Amonson, Paul D wrote:
> > > And this bit doesn't look good. The LICENSE file says:
> > ...
> > > > //* Redistributions in binary form must reproduce the above
> > > > // copyright notice, this list of conditions and the following
> > > > disclaimer // in the documentation and/or other materials provided
> > > > with the // distribution.
> > ...
> > > The second clause essentially says we would have to add a page to our
> > > "documentation and/or other materials" with the contents of the license
> > > file.
> >
> > According to one of Intel’s lawyers, 55 instances of this clause was found
> > when they searched in the PostgreSQL repository. Therefore, I assume that
> > this obligation has either been satisfied or determined not to apply, given
> > that the second BSD clause already appears in the PostgreSQL source tree. I
> > might have misunderstood the concern, but the lawyer believes this is a
> > non-issue. Could you please provide more clarifying details about the
> > concern?
>
> Yes, I can confirm that:
>
> grep -Rl 'Redistributions in binary form must reproduce' . | wc -l
>
> reports 54; file list attached.
I am somewhat embarrassed by this since we made the Intel lawyers find
something that was in our own source code.
First, the "advertizing clause" in the 4-clause license:
3. All advertising materials mentioning features or use of this
software must display the following acknowledgement: This product
includes software developed by the University of California,
Berkeley and its contributors.
and was disavowed by Berkeley on July 22nd, 1999:
https://elrc-share.eu/static/metashare/licences/BSD-3-Clause.pdf
While the license we are concerned about does not have this clause, it
does have:
2. Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following
disclaimer in the documentation and/or other materials provided
with the distribution.
I assume that must also include the name of the copyright holder.
I think that means we need to mention The Regents of the University of
California in our copyright notice, which we do. However several
non-Regents of the University of California copyright holder licenses
exist in our source tree, and accepting this AVX-512 patch would add
another one. Specifically, I see existing entries for:
Aaron D. Gifford
Board of Trustees of the University of Illinois
David Burren
Eric P. Allman
Jens Schweikhardt
Marko Kreen
Sun Microsystems, Inc.
WIDE Project
Now, some of these are these names plus Berkeley, and some are just the
names above.
--
Bruce Momjian https://momjian.us
EDB https://enterprisedb.com
Only you can decide what is important to you.
Re: Proposal for Updating CRC32C with AVX-512 Algorithm.
On Tue, Jun 18, 2024 at 05:14:08PM +, Amonson, Paul D wrote: > > And this bit doesn't look good. The LICENSE file says: > ... > > > //* Redistributions in binary form must reproduce the above > > > // copyright notice, this list of conditions and the following > > > disclaimer // in the documentation and/or other materials provided > > > with the // distribution. > ... > > The second clause essentially says we would have to add a page to our > > "documentation and/or other materials" with the contents of the license > > file. > > According to one of Intel’s lawyers, 55 instances of this clause was found > when they searched in the PostgreSQL repository. Therefore, I assume that > this obligation has either been satisfied or determined not to apply, given > that the second BSD clause already appears in the PostgreSQL source tree. I > might have misunderstood the concern, but the lawyer believes this is a > non-issue. Could you please provide more clarifying details about the concern? Yes, I can confirm that: grep -Rl 'Redistributions in binary form must reproduce' . | wc -l reports 54; file list attached. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you. ./src/include/lib/sort_template.h ./src/tools/pg_bsd_indent/err.c ./src/tools/pg_bsd_indent/lexi.c ./src/tools/pg_bsd_indent/indent_globs.h ./src/tools/pg_bsd_indent/pr_comment.c ./src/tools/pg_bsd_indent/indent.c ./src/tools/pg_bsd_indent/parse.c ./src/tools/pg_bsd_indent/io.c ./src/tools/pg_bsd_indent/indent_codes.h ./src/tools/pg_bsd_indent/err.h ./src/tools/pg_bsd_indent/indent.h ./src/tools/pg_bsd_indent/args.c ./src/backend/utils/mb/wstrcmp.c ./src/backend/utils/mb/wstrncmp.c ./src/common/md5.c ./src/common/sha2_int.h ./src/common/sha1_int.h ./src/common/sha2.c ./src/common/sha1.c ./src/common/md5_int.h ./src/timezone/strftime.c ./src/port/pgmkdirp.c ./src/port/getopt_long.c ./src/port/mkdtemp.c ./src/port/getopt.c ./src/port/snprintf.c ./src/port/bsearch_arg.c ./src/port/inet_aton.c ./contrib/pgcrypto/openssl.c ./contrib/pgcrypto/pgp-pubdec.c ./contrib/pgcrypto/pgp-cfb.c ./contrib/pgcrypto/pgcrypto.h ./contrib/pgcrypto/mbuf.h ./contrib/pgcrypto/pgcrypto.c ./contrib/pgcrypto/crypt-des.c ./contrib/pgcrypto/px-hmac.c ./contrib/pgcrypto/pgp-armor.c ./contrib/pgcrypto/px.c ./contrib/pgcrypto/px-crypt.c ./contrib/pgcrypto/mbuf.c ./contrib/pgcrypto/px-crypt.h ./contrib/pgcrypto/pgp-pgsql.c ./contrib/pgcrypto/pgp-mpi.c ./contrib/pgcrypto/pgp-mpi-openssl.c ./contrib/pgcrypto/pgp-decrypt.c ./contrib/pgcrypto/pgp.h ./contrib/pgcrypto/px.h ./contrib/pgcrypto/pgp-info.c ./contrib/pgcrypto/pgp-pubenc.c ./contrib/pgcrypto/pgp-encrypt.c ./contrib/pgcrypto/pgp-compress.c ./contrib/pgcrypto/pgp-s2k.c ./contrib/pgcrypto/pgp.c ./contrib/pgcrypto/pgp-pubkey.c
Re: PostgreSQL 17 Beta 2 release date & commit freeze
On Tue, Jun 18, 2024 at 12:10:50PM -0400, Jonathan Katz wrote: > > Hi, > > PostgreSQL 17 Beta 2 is planned to be release on June 27, 2024. Please > continue your hard work on closing out open items[1] ahead of the release > and have the fixes targeted for the release committed by June 22, 2024. I am adding markup to the PG 17 release notes and will be finished by this Friday. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Missing docs for new enable_group_by_reordering GUC
This commit added enable_group_by_reordering: commit 0452b461bc4 Author: Alexander Korotkov Date: Sun Jan 21 22:21:36 2024 +0200 Explore alternative orderings of group-by pathkeys during optimization. When evaluating a query with a multi-column GROUP BY clause, we can minimize sort operations or avoid them if we synchronize the order of GROUP BY clauses with the ORDER BY sort clause or sort order, which comes from the underlying query tree. Grouping does not imply any ordering, so we can compare the keys in arbitrary order, and a Hash Agg leverages this. But for Group Agg, we simply compared keys in the order specified in the query. This commit explores alternative ordering of the keys, trying to find a cheaper one. The ordering of group keys may interact with other parts of the query, some of which may not be known while planning the grouping. For example, there may be an explicit ORDER BY clause or some other ordering-dependent operation higher up in the query, and using the same ordering may allow using either incremental sort or even eliminating the sort entirely. The patch always keeps the ordering specified in the query, assuming the user might have additional insights. This introduces a new GUC enable_group_by_reordering so that the optimization may be disabled if needed. Discussion: https://postgr.es/m/7c79e6a5-8597-74e8-0671-1c39d124c9d6%40sigaev.ru Author: Andrei Lepikhov, Teodor Sigaev Reviewed-by: Tomas Vondra, Claudio Freire, Gavin Flower, Dmitry Dolgov Reviewed-by: Robert Haas, Pavel Borisov, David Rowley, Zhihong Yu Reviewed-by: Tom Lane, Alexander Korotkov, Richard Guo, Alena Rybakina It mentions it was added as a GUC to postgresql.conf, but I see no SGML docs for this new GUC value. Would someone please add docs for this? Thanks. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: Proposal for Updating CRC32C with AVX-512 Algorithm.
On Wed, Jun 12, 2024 at 02:08:02PM -0400, Tom Lane wrote: > "Amonson, Paul D" writes: > > I had our OSS internal team, who are experts in OSS licensing, review > > possible conflicts between the PostgreSQL license and the BSD-Clause 3-like > > license for the CRC32C AVX-512 code, and they found no issues. Therefore, > > including the new license into the PostgreSQL codebase should be acceptable. > > Maybe you should get some actual lawyers to answer this type of > question. The Chromium license this code cites is 3-clause-BSD > style, which is NOT compatible: the "advertising" clause is > significant. > > In any case, writing copyright notices that are pointers to > external web pages is not how it's done around here. We generally > operate on the assumption that the Postgres source code will > outlive any specific web site. Dead links to incidental material > might be okay, but legally relevant stuff not so much. Agreed. The licenses are compatible in the sense that they can be combined to create a unified work, but they cannot be combined without modifying the license of the combined work. You would need to combine the Postgres and Chrome license for this, and I highly doubt we are going to be modifying the Postgres for this. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Wed, Jun 5, 2024 at 11:46:17PM +0100, Dean Rasheed wrote: > On Thu, 9 May 2024 at 05:03, Bruce Momjian wrote: > > > > I have committed the first draft of the PG 17 release notes; you can > > see the results here: > > > > https://momjian.us/pgsql_docs/release-17.html > > I noticed a couple more things. This item: > > Add functions to convert integers to hex and binary strings > > should read: > > Add functions to convert integers to binary and octal strings > > > The "Improve psql tab completion" item should include this commit: > > Author: Michael Paquier > 2024-05-01 [2800fbb2b] Add tab completion for EXPLAIN (MEMORY|SERIALIZE) > > and credit Jian He. Agreed, attached patch applied. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you. diff --git a/doc/src/sgml/release-17.sgml b/doc/src/sgml/release-17.sgml index 1131f2aab51..93bc7408b5b 100644 --- a/doc/src/sgml/release-17.sgml +++ b/doc/src/sgml/release-17.sgml @@ -1685,7 +1685,7 @@ Author: Nathan Bossart -Add functions to convert integers to hex and binary strings (Eric Radman, Nathan Bossart) +Add functions to convert integers to binary and octal strings (Eric Radman, Nathan Bossart) @@ -2000,11 +2000,13 @@ Author: Masahiko Sawada 2024-04-08 [304b6b1a6] Add more tab completion support for ALTER DEFAULT PRIVIL Author: Alexander Korotkov 2024-04-30 [60ae37a8b] Add tab completion for partition MERGE/SPLIT operations +Author: Michael Paquier +2024-05-01 [2800fbb2b] Add tab completion for EXPLAIN (MEMORY|SERIALIZE) --> -Improve psql tab completion (Dagfinn Ilmari Mannsåker, Gilles Darold, Christoph Heiss, Steve Chavez, Vignesh C, Pavel Borisov) +Improve psql tab completion (Dagfinn Ilmari Mannsåker, Gilles Darold, Christoph Heiss, Steve Chavez, Vignesh C, Pavel Borisov, Jian He)
Re: Partial aggregates pushdown
On Wed, Jun 5, 2024 at 08:19:04AM +0300, Alexander Pyhalov wrote: > > * Passes unsafe binary data from the foreign server. > > > > Can someone show me where that last item is in the patch, and why can't > > we just pass back values cast to text? > > In finalize_aggregate() when we see partial aggregate with > peragg->aggref->aggtranstype = INTERNALOID > we call aggregate's serialization function and return it as bytea. > > The issue is that this internal representation isn't guaranteed to be > compatible between servers > of different versions (or architectures?). So, likely, we instead should > have called some export function for aggregate > and later - some import function on postgres_fdw side. It doesn't matter > much, what this export function > generates - text, json or some portable binary format, > 1) export/import functions should just "understand" it, > 2) it should be a stable representation. Okay, so looking at the serialization output functions already defined, I see many zeros, which I assume means just the base data type, and eight more: SELECT DISTINCT aggserialfn from pg_aggregate WHERE aggserialfn::oid != 0; aggserialfn --- numeric_avg_serialize string_agg_serialize array_agg_array_serialize numeric_serialize int8_avg_serialize array_agg_serialize interval_avg_serialize numeric_poly_serialize I realize we need to return the sum and count for average, so that makes sense. So, we need import/export text representation for the partial aggregate mode for these eight, and call the base data type text import/export functions for the zero ones when in this mode? -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: Partial aggregates pushdown
On Wed, Jun 5, 2024 at 12:14:45AM +, fujii.y...@df.mitsubishielectric.co.jp wrote: > I will add sufficient document and comments to the next patch as Bruce and > Robert said. Great, I am available to help improve the documentation. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: Partial aggregates pushdown
On Mon, May 27, 2024 at 09:30:59PM +, fujii.y...@df.mitsubishielectric.co.jp wrote: > Hi Mr. Pyhalov. > > Sorry for the late reply. > Thank you for your modification and detailed review. > I attach a fixed patch, have been not yet rebased. I know this patch was discussed at the Vancouver conference. What are the open issues? I know of several: * add tests that were requested by Fujii-san and now posted by Alexander Pyhalov * Where is the documentation? I know the original patch had some, and I improved it, but it seems to be missing. * Passes unsafe binary data from the foreign server. Can someone show me where that last item is in the patch, and why can't we just pass back values cast to text? -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Tue, May 28, 2024 at 02:44:28PM +1200, David Rowley wrote: > On Sun, 26 May 2024 at 15:57, Bruce Momjian wrote: > > Agreed. I changed it to: > > > > Allow btree indexes to more efficiently find a set of values, such > > as > > those supplied by IN subqueries > > > > Is that good? > > I think this needs further adjustment. An "IN subquery" is an IN > clause which contains a subquery. As far as I understand it, > 5bf748b86 does nothing to improve those. It's there to improve IN with > a set of values such as IN(1,2,3). > > Maybe "IN subqueries" can be replaced with "an SQL IN clause". Okay, I went with: Allow btree indexes to more efficiently find a set of values, such as those supplied by IN clauses using constants (Peter Geoghegan, Matthias van de Meent) -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Sun, May 26, 2024 at 10:10:00AM +0200, Álvaro Herrera wrote:
> On 2024-May-25, Bruce Momjian wrote:
>
> > On Thu, May 23, 2024 at 01:22:51PM +0200, Álvaro Herrera wrote:
>
> > > Can we make them a single item? Maybe something like
> > >
> > > : Improve reset routines for shared statistics (Atsushi Torikoshi,
> > > Bharath Rupireddy)
> > > :
> > > : Resetting all shared statistics can now be done with
> > > : pg_stat_reset_shared() or pg_stat_reset_shared(NULL), while SLRU
> > > : statistics can now be reset with pg_stat_reset_shared('slru'),
> > > : pg_stat_reset_slru() and pg_stat_reset_slru(NULL).
> >
> > Andres already suggested improvement for this, and I posted the applied
> > patch. Can you see if that is good or can be improved? Thanks.
>
> Yeah, looks good, thanks.
Wow, that's great. My head started to spin trying to make sense of how
those three entries connected. Glad we were able to condense them, and
the new result is easier to read.
--
Bruce Momjian https://momjian.us
EDB https://enterprisedb.com
Only you can decide what is important to you.
Re: AIX support
On Thu, May 23, 2024 at 07:03:20PM +0300, Heikki Linnakangas wrote: > > Can you provide some more details on the expectations here? > > Smallest possible patch that makes Postgres work on AIX again. > > Perhaps start with the patch you posted yesterday, but remove hunks from it > one by one, to see which ones are still needed. Yes, bingo, that is exactly what needs to be done, and for the minimal compiler, gcc, and the most recently supported versions of AIX. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Thu, May 23, 2024 at 08:19:15PM -0400, Peter Geoghegan wrote: > On Wed, May 22, 2024 at 6:50 PM Bruce Momjian wrote: > > Agreed, patch applied, thanks. > > The item for my commit 5bf748b8 currently reads: > > "Allow btree indexes to more efficiently find array matches" > > I think that this isn't likely to mean much to most users. It seems > like it'd be a lot clearer if the wording was more in line with the > beta1 announcement, which talks about the work as an enhancement to > index scans that use an IN ( ) condition. Specifically referencing > IN() (as opposed to something about arrays or array conditions) is > likely to make the item much more understandable. > > Referencing array matches makes me think of a GIN index on an array > column. While ScalarArrayOps do use an array under the cover, that's > mostly an implementation detail. At least it is to users that > exclusively use IN(), likely the majority (that's the SQL standard > syntax). > > For context, the Postgres 9.2 release notes described the feature that > my work directly built on as follows: > > "Allow indexed_col op ANY(ARRAY[...]) conditions to be used in plain > index scans and index-only scans" > > This was a very accurate description of this earlier work. Similar > wording could be used now, but that doesn't seem great to me either. > Simply because this wording also doesn't reference IN() conditions in > index quals. Agreed. I changed it to: Allow btree indexes to more efficiently find a set of values, such as those supplied by IN subqueries Is that good? -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Thu, May 23, 2024 at 04:54:28PM -0300, Marcos Pegoraro wrote: > • Rename SLRU columns in system view pg_stat_slru (Alvaro Herrera) > > The column names accepted by pg_stat_slru_rest() are also changed. > > Is pg_stat_slru_rest() correct ? Oops, typo, fixed, thanks. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Thu, May 23, 2024 at 01:22:51PM +0200, Álvaro Herrera wrote:
> Hello,
>
> Regarding this item
>
> : Allow the SLRU cache sizes to be configured (Andrey Borodin, Dilip Kumar)
> :
> : The new server variables are commit_timestamp_buffers,
> : multixact_member_buffers, multixact_offset_buffers, notify_buffers,
> : serializable_buffers, subtransaction_buffers, and transaction_buffers.
>
> I hereby request to be listed as third author of this feature.
>
> Also, I'd like to suggest to make it more verbose, as details might be
> useful to users. Mention that scalability is improved, because
> previously we've suggested to recompile with larger #defines, but to be
> cautious because values too high degrade performance. Also mention the
> point that some of these grow with shared_buffers is user-visible enough
> that it warrants an explicit mention. How about like this:
>
> : Allow the SLRU cache sizes to be configured and improve performance of
> : larger caches
> : (Andrey Borodin, Dilip Kumar, Álvaro Herrera)
> :
> : The new server variables are commit_timestamp_buffers,
> : multixact_member_buffers, multixact_offset_buffers, notify_buffers,
> : serializable_buffers, subtransaction_buffers, and transaction_buffers.
> : commit_timestamp_buffers, transaction_buffers and
> : subtransaction_buffers scale up automatically with shared_buffers.
Yes, I like that, patch applied.
> These three items
>
> : Allow pg_stat_reset_shared() to reset all shared statistics (Atsushi
> Torikoshi)
> :
> : This is done by passing NULL.
> :
> : Allow pg_stat_reset_shared('slru') to clear SLRU statistics (Atsushi
> Torikoshi)
> :
> : Now pg_stat_reset_shared(NULL) also resets SLRU statistics.
> :
> : Allow pg_stat_reset_slru() to reset all SLRU statistics (Bharath Rupireddy)
> :
> : The command pg_stat_reset_slru(NULL) already did this.
>
> seem a bit repetitive. (I think the first one is also wrong, because it
> says you have to pass NULL, but in reality you can also not give an
> argument and it works.) Can we make them a single item? Maybe
> something like
>
> : Improve reset routines for shared statistics (Atsushi Torikoshi, Bharath
> Rupireddy)
> :
> : Resetting all shared statistics can now be done with
> : pg_stat_reset_shared() or pg_stat_reset_shared(NULL), while SLRU
> : statistics can now be reset with pg_stat_reset_shared('slru'),
> : pg_stat_reset_slru() and pg_stat_reset_slru(NULL).
Andres already suggested improvement for this, and I posted the applied
patch. Can you see if that is good or can be improved? Thanks.
--
Bruce Momjian https://momjian.us
EDB https://enterprisedb.com
Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Fri, May 24, 2024 at 11:23:29AM -0700, Andres Freund wrote:
> Hi,
>
> On 2024-05-22 18:33:03 -0400, Bruce Momjian wrote:
> > On Tue, May 21, 2024 at 09:40:28AM -0700, Andres Freund wrote:
> > > On 2024-05-18 11:13:54 -0400, Bruce Momjian wrote:
> > > I agree keeping things reasonably short is important. But I don't think
> > > you're
> > > evenly applying it as a goal.
> > >
> > > Just skimming the notes from the end, I see
> > > - an 8 entries long pg_stat_statements section
> >
> > What item did you want to remove? Those are all user-visible changes.
>
> My point here was not that we necessarily need to remove those, but that their
> impact to users is smaller than many of the performance impacts you disregard.
I liked all your detailed suggestions so applied the attached patch.
--
Bruce Momjian https://momjian.us
EDB https://enterprisedb.com
Only you can decide what is important to you.
diff --git a/doc/src/sgml/release-17.sgml b/doc/src/sgml/release-17.sgml
index 891678cc94b..1e65e99f2b2 100644
--- a/doc/src/sgml/release-17.sgml
+++ b/doc/src/sgml/release-17.sgml
@@ -628,45 +628,20 @@ Relevant columns have been removed from pg_stat_bgwriter and added to this new s
-
-
-
-Allow pg_stat_reset_shared() to reset all shared statistics (Atsushi Torikoshi)
-
-
-
-This is done by passing NULL.
-
-
-
-
-
-
-
-Allow pg_stat_reset_shared('slru') to clear SLRU statistics (Atsushi Torikoshi)
-
-
-
-Now pg_stat_reset_shared(NULL) also resets SLRU statistics.
-
-
-
-
-Allow pg_stat_reset_slru() to reset all SLRU statistics (Bharath Rupireddy)
+Improve control over resetting statistics (Atsushi Torikoshi, Bharath Rupireddy)
-The command pg_stat_reset_slru(NULL) already did this.
+Allow pg_stat_reset_shared() (with no arguments) and pg_stat_reset_shared(NULL) to reset all shared statistics.
+Allow pg_stat_reset_shared('slru') and pg_stat_reset_slru() (with no arguments) to reset SLRU statistics, which was already possible with pg_stat_reset_slru(NULL).
@@ -784,21 +759,22 @@ Add server variable trace_connection_negotiation to allow debugging of connectio
-Add per-table GRANT permission MAINTAIN to control maintenance operations (Nathan Bossart)
+Allow granting the right to perform maintenance operations (Nathan Bossart)
-The operations are VACUUM, ANALYZE, REINDEX, REFRESH MATERIALIZED VIEW, CLUSTER, and LOCK TABLE.
+The permission can be granted on a per-table basis using the MAINTAIN privilege and on a per-role basis via the pg_maintain predefined role. Permitted operations are VACUUM, ANALYZE,
+REINDEX, REFRESH MATERIALIZED VIEW, CLUSTER, and LOCK TABLE.
@@ -2234,45 +2210,19 @@ Allow reindexdb --index to process indexes from different tables in parallel (Ma
-
-
-
-Allow reindexdb to process objects in all databases matching a pattern (Nathan Bossart)
-
-
-
-Specifically, --all can now be used with --table, --schema, --index, and --system.
-
-
-
-
-
-
-
-Allow vacuumdb to process objects in all databases matching a pattern (Nathan Bossart)
-
-
-
-Specifically, --all can now be used with --table, --schema, and --exclude-schema.
-
-
-
-
-Allow clusterdb to process objects in all databases matching a pattern (Nathan Bossart)
+Allow reindexdb, vacuumdb, and clusterdb to process objects in all databases matching a pattern (Nathan Bossart)
-Specifically, --all can now be used with --table.
+The new option --all controls this behavior.
@@ -2550,28 +2500,6 @@ This value is used by the optimizer.
-
-
-
-
-Create custom wait events for postgres_fdw (Masahiro Ikeda)
-
-
-
-
-
-
-
-Create custom wait events for dblink (Masahiro Ikeda)
-
-
-
Allow extensions to define custom wait events (Masahiro Ikeda)
+
+
+Custom wait events have been added to postgres_fdw and dblink.
+
Re: First draft of PG 17 release notes
On Fri, May 24, 2024 at 10:50:28AM -0700, Andres Freund wrote: > Hi, > > On 2024-05-23 23:27:04 -0400, Bruce Momjian wrote: > > On Thu, May 23, 2024 at 11:11:10PM -0400, Tom Lane wrote: > > > Bruce Momjian writes: > > > I am not sure Bruce that you realize that your disregard for > > > performance improvements is shared by nobody. Arguably, > > > performance is 90% of what we do these days, and it's also > > > 90% of what users care about. > > > > Please stop saying I don't document performance. I have already > > explained enough which performance items I choose. Please address my > > criteria or suggest new criteria. > > Bruce, just about everyone seems to disagree with your current approach. And > not just this year, this has been a discussion in most if not all release note > threads of the last few years. > > People, including me, *have* addressed your criteria, but you just waved those > concerns away. It's hard to continue discussing criteria when it doesn't at > all feel like a conversation. > > In the end, these are patches to the source code, I don't think you can just > wave away widespread disagreement with your changes. That's not how we do > postgres development. Well, let's start with a new section for PG 17 that lists these. Is it 20 items, 50, or 150? I have no idea, but without the user-visible filter, I am unable to determine what not-included performance features are worthy of the release notes. Can someone do that? There is no reason other committers can't change the release notes. Yes, I realize we are looking for a consistent voice, but the new section can probably have its own style, and I can make adjustments if desired. Also, I think this has gone unaddressed so long because if we skip a user-visible change, users complain, but I don't remember anyone complaining about skipped performance changes. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Thu, May 23, 2024 at 11:11:10PM -0400, Tom Lane wrote: > Bruce Momjian writes: > > On Thu, May 23, 2024 at 02:27:07PM +1200, David Rowley wrote: > >> I also don't agree these should be left to "Source code" section. I > >> feel that section is best suited for extension authors who might care > >> about some internal API change. I'm talking of stuff that makes some > >> user queries possibly N times (!) faster. Surely "Source Code" isn't > >> the place to talk about that? > > > I said a new section "after our 'Source code' section," not in the > > source code section. > > Surely, if we make this a separate section, it would come before > 'Source code'? > > I am not sure Bruce that you realize that your disregard for > performance improvements is shared by nobody. Arguably, > performance is 90% of what we do these days, and it's also > 90% of what users care about. Please stop saying I don't document performance. I have already explained enough which performance items I choose. Please address my criteria or suggest new criteria. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Thu, May 23, 2024 at 02:27:07PM +1200, David Rowley wrote: > On Thu, 23 May 2024 at 14:01, Bruce Momjian wrote: > > > > On Thu, May 23, 2024 at 01:34:10PM +1200, David Rowley wrote: > > > What is the best way to communicate this stuff so it's easily > > > identifiable when you parse the commit messages? > > > > This is why I think we need an "Internal Performance" section where we > > can be clear about simple scaling improvements that might have no > > user-visible explanation. I would suggest putting it after our "Source > > code" section. > > hmm, but that does not really answer my question. There's still a > communication barrier if you're parsing the commit messages are > committers don't clearly state some performance improvement numbers > for the reason I stated. For a case where O(N^2) become O(N), we might not even know the performance change since it is a micro-optimization. That is why I suggested we call it "Internal Performance". > I also don't agree these should be left to "Source code" section. I > feel that section is best suited for extension authors who might care > about some internal API change. I'm talking of stuff that makes some > user queries possibly N times (!) faster. Surely "Source Code" isn't > the place to talk about that? I said a new section "after our 'Source code' section," not in the source code section. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Thu, May 23, 2024 at 01:34:10PM +1200, David Rowley wrote: > On Thu, 23 May 2024 at 10:04, Bruce Momjian wrote: > > You might have seen in this thread, I do record commits that speed up > > workloads that are user-visible, or specifically make new workloads > > possible. I assume that covers the items above, though I have to > > determine this from the commit message. > > It sometimes is hard to write something specific in the commit message > about the actual performance increase. > > For example, if a commit removes an O(N log2 N) algorithm and replaces > it with an O(1), you can't say there's an X% increase in performance > as the performance increase depends on the value of N. > > Jelte did call me out for not mentioning enough detail about the > performance in c4ab7da60, but if I claimed any % of an increase, it > would have been specific to some workload. > > What is the best way to communicate this stuff so it's easily > identifiable when you parse the commit messages? This is why I think we need an "Internal Performance" section where we can be clear about simple scaling improvements that might have no user-visible explanation. I would suggest putting it after our "Source code" section. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Wed, May 22, 2024 at 09:25:41PM +0900, torikoshia wrote:
> Thanks for working on this as always.
>
>
> Allow pg_stat_reset_shared("slru") to clear SLRU statistics (Atsushi
> Torikoshi)
>
>
> Considering someone may copy and paste this, 'slru' is better than "slru",
> isn't it?
> I also found an older release note[1] used single quotes for this like:
>
> Add pg_stat_reset_shared('bgwriter') to reset the cluster-wide shared
> statistics for the background writer (Greg Smith)
Agreed, patch applied, thanks.
--
Bruce Momjian https://momjian.us
EDB https://enterprisedb.com
Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Wed, May 22, 2024 at 11:29:06AM +0900, Masahiko Sawada wrote: > I found a typo: > > s/pg_statstatement/pg_stat_statement/ > > I've attached a patch to fix it. Agreed, applied, thanks. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Tue, May 21, 2024 at 11:20:02AM +0900, Amit Langote wrote: > Thanks Bruce for working on this as always. > > Failed to notice when I read the notes before: > > > > Add SQL/JSON constructor functions JSON(), JSON_SCALAR(), and > JSON_SERIALIZE() (Amit Langote) > > > > Should be: > > > > Add SQL/JSON constructor functions JSON(), JSON_SCALAR(), and > JSON_SERIALIZE() (Nikita Glukhov, Teodor Sigaev, Oleg Bartunov, > Alexander Korotkov, Andrew Dunstan, Amit Langote) > > Thanks, applied. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Mon, May 20, 2024 at 11:48:09AM -0700, Jeff Davis wrote: > On Sat, 2024-05-18 at 17:51 -0400, Bruce Momjian wrote: > > Okay, I went with the attached applied patch. Adjustments? > > I think it should have more emphasis on the actual new feature: a > platform-independent builtin collation provider and the C.UTF-8 locale. > > The user can look at the documentation for comparison with libc. Okay, changed with the attached applied patch. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you. diff --git a/doc/src/sgml/release-17.sgml b/doc/src/sgml/release-17.sgml index 3025f31f8a0..e381dbe5d96 100644 --- a/doc/src/sgml/release-17.sgml +++ b/doc/src/sgml/release-17.sgml @@ -885,11 +885,11 @@ Author: Jeff Davis -Create "builtin" collation provider similar to libc's C locale (Jeff Davis) +Add a builtin platform-independent collation provider (Jeff Davis) -While its C locale is similar but independent of libc, its C.UTF-8 locale sorts by Unicode code points and has Unicode-based case folding. +This supports "C" and "C.UTF-8" collations.
Re: First draft of PG 17 release notes
On Tue, May 21, 2024 at 09:40:28AM -0700, Andres Freund wrote:
> Hi,
>
> On 2024-05-18 11:13:54 -0400, Bruce Momjian wrote:
> > Please see the email I just posted. There are three goals we have to
> > adjust for:
> >
> > 1. short release notes so they are readable
> > 2. giving people credit for performance improvements
> > 3. showing people Postgres cares about performance
> >
> > I would like to achieve 2 & 3 without harming #1. My experience is if I
> > am reading a long document, and I get to a section where I start to
> > wonder, "Why should I care about this?", I start to skim the rest of
> > the document.
>
> I agree keeping things reasonably short is important. But I don't think you're
> evenly applying it as a goal.
>
> Just skimming the notes from the end, I see
> - an 8 entries long pg_stat_statements section
What item did you want to remove? Those are all user-visible changes.
> - multiple entries about "Create custom wait events for ..."
Well, those are all in different sections, so how can they be merged,
unless I create a "wait event section", I guess.
> - three entries about adding --all to {reindexdb,vacuumdb,clusterdb}.
The problem with merging these is that the "Specifically, --all can now
be used with" is different for all three of them.
> - an entry about adding long options to pg_archivecleanup
Well, that is a user-visible change. Should it not be listed?
> - two entries about grantable maintenance rights, once via pg_maintain, once
> per-table
Well, one is a GRANT and another is a role, so merging them seemed like
it would be too confusing.
> - separate entries about pg_stat_reset_slru(), pg_stat_reset_shared("slru"),
They are different functions with different detail text.
--
Bruce Momjian https://momjian.us
EDB https://enterprisedb.com
Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Tue, May 21, 2024 at 02:26:15PM -0400, Melanie Plageman wrote: > In Postgres development, we break larger projects into smaller ones > and then those smaller projects into multiple individual commits. Each > commit needs to stand alone and each subproject needs to have a > defensible benefit. One thing that is harder with performance-related > work than non-performance feature work is that there isn't always a > final "turn it on" commit. For example, let's say you are adding a new > view that tracks new stats of some kind. You do a bunch of refactoring > and small subprojects to make it possible to add the view. Then the > final commit that actually creates the view has obvious user value to > whoever is reading the log. For performance features, it doesn't > always work like this. > > For the vacuum WAL volume reduction, there were a bunch of smaller > projects throughout the last development year that I worked on that > were committed by different people and with different individual > benefits. Some changes caused vacuum to do less visibility checks (so > less CPU usage), some changed WAL format in a way that saves some > space, and some, like the commit you mention, make vacuum emit less > WAL. That commit by itself doesn't contain all of the user benefits of > the whole project. I couldn't think of a good place to list all of the > commits together that were part of the same project. Perhaps you could > argue that they were not in fact part of the same project and instead > were just small individual changes -- none of which are individually > worth including in the release notes. I try and group them, but I am sure imperfectly. It is very true that infrastucture changes that enable later commits are often missed. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Tue, May 21, 2024 at 01:50:58PM -0400, Robert Haas wrote:
> On Tue, May 21, 2024 at 12:27 PM Andres Freund wrote:
> > To me that's the "General Performance" section. If somebody reading the
> > release notes doesn't care about performance, they can just skip that
> > section
> > ([1]). I don't see why we wouldn't want to include the same level of detail
> > as for other changes.
>
> I'm relatively sure that we've had this argument in previous years and
> essentially everyone but Bruce has agreed with the idea that
> performance changes ought to be treated the same as any other kind of
> improvement. The difficulty is that Bruce is the one doing the release
> notes. I think it might help if someone were willing to prepare a
> patch showing what they think specifically should be changed. Or maybe
> Bruce would be willing to provide a list of all of the performance
> improvements he doesn't think are worth release-noting or isn't sure
> how to release-note, and someone else can then have a go at them.
Well, developers do ask why their individual commits were not listed,
and I repeat the same thing, as I have done in this thread multiple
times. You can probably look over this thread to find them, or in
previous years.
To be clear, it is performance improvements that don't have user-visible
changes or enable new workloads that I skip listing. I will also note
that don't remember any user ever finding a performance boost, and then
coming to use and asking why we didn't list it --- this release note
review process seems to add all of those.
Maybe adding a section called "Internal Performance". Here is our
General Performance contents:
E.1.3.1.3. General Performance
Allow vacuum to more efficiently remove and freeze tuples (Melanie
Plageman)
WAL traffic caused by vacuum is also more compact.
Allow vacuum to more efficiently store tuple references (Masahiko
Sawada, John Naylor)
Additionally, vacuum is no longer silently limited to one gigabyte
of memory when maintenance_work_mem or autovacuum_work_mem are higher.
Optimize vacuuming of relations with no indexes (Melanie Plageman)
Increase default vacuum_buffer_usage_limit to 2MB (Thomas Munro)
Improve performance when checking roles with many memberships
(Nathan Bossart)
Improve performance of heavily-contended WAL writes (Bharath
Rupireddy)
Improve performance when transferring large blocks of data to a
client (Melih Mutlu)
Allow the grouping of file system reads with the new system variable
io_combine_limit (Thomas Munro, Andres Freund, Melanie Plageman, Nazir
Bilal Yavuz)
Do we think more user-invisible changes should be in there?
--
Bruce Momjian https://momjian.us
EDB https://enterprisedb.com
Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Tue, May 21, 2024 at 09:51:09AM -0700, Andres Freund wrote: > Hi, > > On 2024-05-21 09:27:20 -0700, Andres Freund wrote: > > Also, the release notes are also not just important to users. I often go > > back > > and look in the release notes to see when some some important change was > > made, > > because sometimes it's harder to find it in the git log, due to sheer > > volume. And even just keeping up with all the changes between two releases > > is > > hard, it's useful to be able to read the release notes and see what > > happened. > > > > [...] > > > > [1] I've wondered if we should have one more level of TOC on the release > > note > > page, so it's easier to jump to specific sections. > > Which reminds me: Eventually I'd like to add links to the most important > commits related to release note entries. We already do much of the work of > building that list of commits for each entry. That'd allow a reader to find > more details if interested. Yes, it would be cool if they could mouse-over a graphic next to each release note item to get a popup to the commits. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Tue, May 21, 2024 at 09:27:20AM -0700, Andres Freund wrote: > On 2024-05-18 10:59:47 -0400, Bruce Momjian wrote: > > I agree the impact of performance improvements are often greater than > > the average release note item. However, if people expect Postgres to be > > faster, is it important for them to know _why_ it is faster? > > Yes, it very often is. Performance improvements typically aren't "make > everything 3% faster", they're more "make this special thing 20% > faster". Without know what got faster, users don't know if > a) the upgrade will improve their production situation > b) they need to change something to take advantage of the improvement You might have seen in this thread, I do record commits that speed up workloads that are user-visible, or specifically make new workloads possible. I assume that covers the items above, though I have to determine this from the commit message. > > On the flip side, a performance improvement that makes everything 10% > > faster has little behavioral change for users, and in fact I think we > > get ~6% faster in every major release. > > I cannot recall many "make everything faster" improvements, if any. > > And even if it's "make everything faster" - that's useful for users to know, > it might solve their production problem! It's also good for PR. Again, it is down to having three goals for the release notes, and #1 is having it readable/short, and 2 & 3 are for PR and acknowledging authors. > Also, the release notes are also not just important to users. I often go back > and look in the release notes to see when some some important change was made, > because sometimes it's harder to find it in the git log, due to sheer > volume. And even just keeping up with all the changes between two releases is > hard, it's useful to be able to read the release notes and see what happened. Well, I would say we need some _other_ way to record and perhaps advertise such changes. > > > For another, it's also very frustrating for developers that focus on > > > performance. The reticence to note their work, while noting other, far > > > smaller, things in the release notes, pretty much tells us that our work > > > isn't > > > valued. > > > > Yes, but are we willing to add text that every user will have to read > > just for this purpose? > > Of course it's a tradeoff. We shouldn't verbosely note down every small > changes just because of the recognition, that'd make the release notes > unreadable. And it'd just duplicate the commit log. But that's not the same as > defaulting to not noting performance improvements, even if the performance > improvement is more impactful than many other features that are noted. Again, see above, I do mention performance improvements, but they have to be user-visible or enable new workloads. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Mon, May 20, 2024 at 02:47:28PM -0400, Melanie Plageman wrote: > On Mon, May 20, 2024 at 9:37 AM Bruce Momjian wrote: > > > > On Mon, May 20, 2024 at 01:23:02PM +0700, John Naylor wrote: > > > Hi Bruce, thanks for doing this again! > > > > > > I'm a bit late to this discussion -- there's been a bit of churn in > > > the vacuum items, and some streams got crossed along the way. I've > > > attached an attempt to rectify this. > > > > Agreed, patch applied, thanks. > > If "Allow vacuum to more efficiently remove and freeze tuples" stays > in the release notes, I would add Heikki's name. He wasn't listed as a > co-author on all of the commits that were part of this feature, but he > made a substantial investment in it and should be listed, IMO. Patch > attached. Thanks, patch applied. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Mon, May 20, 2024 at 02:35:37PM -0400, Melanie Plageman wrote: > On Sat, May 18, 2024 at 11:13 AM Bruce Momjian wrote: > > Please see the email I just posted. There are three goals we have to > > adjust for: > > > > 1. short release notes so they are readable > > 2. giving people credit for performance improvements > > 3. showing people Postgres cares about performance > > I agree with all three of these goals. I would even add to 3 "show > users Postgres is addressing their performance complaints". That in > particular makes me less excited about having a "generic performance > release note item saying performance has been improved in the > following areas" (from your other email). I think that describing the > specific performance improvements is required to 1) allow users to > change expectations and configurations to take advantage of the > performance enhancements 2) ensure users know that their performance > concerns are being addressed. Well, as you can see, doing #2 & #3 works against accomplishing #1. > > I would like to achieve 2 & 3 without harming #1. My experience is if I > > am reading a long document, and I get to a section where I start to > > wonder, "Why should I care about this?", I start to skim the rest of > > the document. I am particularly critical if I start to wonder, "Why > > does the author _think_ I should care about this?" becasue it feels like > > the author is writing for him/herself and not the audience. > > I see what you are saying. We don't want to just end up with the whole > git log in the release notes. However, we know that not all users will > care about the same features. As someone said somewhere else in this > thread, presumably hackers spend time on features because some users > want them. I think we need as a separate section about performance improvements that don't affect specific workloads. Peter Eisentraut created an Acknowledgements section at the bottom of the release notes, similar to #2 above, so maybe someone else can add a performance internals section too. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Mon, May 20, 2024 at 01:23:02PM +0700, John Naylor wrote: > Hi Bruce, thanks for doing this again! > > I'm a bit late to this discussion -- there's been a bit of churn in > the vacuum items, and some streams got crossed along the way. I've > attached an attempt to rectify this. Agreed, patch applied, thanks. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: commitfest.postgresql.org is no longer fit for purpose
On Sun, May 19, 2024 at 03:18:11PM -0400, Tom Lane wrote: > * Another reason for things sitting a long time is that they're too > big to review without an unreasonable amount of effort. We should > encourage authors to break large patches into smaller stepwise > refinements. It may seem like that will result in taking forever > to reach the end goal, but dropping a huge patchset on the community > isn't going to give speedy results either. I think it is sometimes hard to incrementally apply patches if the long-term goal isn't agreed or know to be achievable. > * Before starting this thread, Robert did a lot of very valuable > review of some individual patches. I think what prompted him to > start the thread was the realization that he'd only made a small > dent in the problem. Maybe we could divide and conquer: get a > dozen-or-so senior contributors to split up the list of pending > patches and then look at each one with an eye to what needs to > happen to move it along (*not* to commit it right away, although > in some cases maybe that's the thing to do). It'd be great if > that could happen just before each commitfest, but that's probably > not practical with the current patch volume. What I'm thinking > for the moment is to try to make that happen once a year or so. For me, if someone already knows what the blocker is, it saves me a lot of time if they can state that somewhere. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Sun, May 19, 2024 at 03:53:38PM +1200, David Rowley wrote: > On Sun, 19 May 2024 at 02:40, Bruce Momjian wrote: > > > > On Thu, May 16, 2024 at 03:35:17PM +1200, David Rowley wrote: > > > "Additionally, vacuum no longer silently imposes a 1GB tuple reference > > > limit even when maintenance_work_mem or autovacuum_work_mem are set to > > > higher values" > > > Slightly adjusted wording patch attached and applied. > > Thanks for adjusting. > > It's a minor detail, but I'll mention it because you went to the > effort to adjust it away from what I'd written... True. > I didn't make a random choice to use "or" between the two GUCs. > Changing it to "and", IMO, isn't an improvement. Using "and" implies > that the silent limited was only imposed when both of these GUCs were > set >= 1GB. That's not true. For the case we're talking about here, if > autovacuum_work_mem is set to anything apart from -1 then the value of > maintenance_work_mem does not matter. Okay, changed to "or". -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: commitfest.postgresql.org is no longer fit for purpose
On Fri, May 17, 2024 at 11:44:40AM -0400, Melanie Plageman wrote: > So, anyway, I'd argue that we need a parking lot for patches which we > all agree are important and have a path forward but need someone to do > the last 20-80% of the work. To avoid this being a dumping ground, > patches should _only_ be allowed in the parking lot if they have a > clear path forward. Patches which haven't gotten any interest don't go > there. Patches in which the author has clearly not addressed feedback > that is reasonable for them to address don't go there. These are > effectively community TODOs which we agree need to be done -- if only > someone had the time. When I am looking to commit something, I have to consider: * do we want the change * are there concerns * are the docs good * does it need tests * is it only a proof-of-concept When people review commit fest entries, they have to figure out what is holding the patch back from being complete, so they have to read the thread from the beginning. Should there be a clearer way in the commit fest app to specify what is missing? -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Fri, May 17, 2024 at 01:30:03PM -0700, Jeff Davis wrote: > On Thu, 2024-05-09 at 00:03 -0400, Bruce Momjian wrote: > > I have committed the first draft of the PG 17 release notes; you can > > see the results here: > > > > https://momjian.us/pgsql_docs/release-17.html > > For this item: > > Create a "builtin" collation provider similar to libc's C > locale (Jeff Davis) > > It uses a "C" locale which is identical but independent of > libc, but it allows the use of non-"C" collations like "en_US" > and "C.UTF-8" with the "C" locale, which libc does not. MORE? > > I suggest something more like: > > New, platform-independent "builtin" collation > provider. (Jeff Davis) > > Currently, it offers the "C" and "C.UTF-8" locales. The > "C.UTF-8" locale combines stable and fast code point order > collation with Unicode character semantics. Okay, I went with the attached applied patch. Adjustments? -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you. diff --git a/doc/src/sgml/release-17.sgml b/doc/src/sgml/release-17.sgml index 0bc1c9a14ad..5d2fcd7d7de 100644 --- a/doc/src/sgml/release-17.sgml +++ b/doc/src/sgml/release-17.sgml @@ -886,11 +886,11 @@ Author: Jeff Davis -Create a "builtin" collation provider similar to libc's C locale (Jeff Davis) +Create "builtin" collation provider similar to libc's C locale (Jeff Davis) -It uses a "C" locale which is identical but independent of libc, but it allows the use of non-"C" collations like "en_US" and "C.UTF-8" with the "C" locale, which libc does not. MORE? +While its C locale is similar but independent of libc, its C.UTF-8 locale sorts by Unicode code points and has Unicode-based case folding.
Re: First draft of PG 17 release notes
On Fri, May 17, 2024 at 09:22:59PM +0800, jian he wrote: > On Thu, May 9, 2024 at 12:04 PM Bruce Momjian wrote: > > > > I have committed the first draft of the PG 17 release notes; you can > > see the results here: > > > > https://momjian.us/pgsql_docs/release-17.html > > > > It will be improved until the final release. The item count is 188, > > which is similar to recent releases: > > > > This thread mentioned performance. > actually this[1] refactored some interval aggregation related functions, > which will make these two aggregate function: avg(interval), sum(interval) > run faster, especially avg(interval). > see [2]. > well, I guess, this is a kind of niche performance improvement to be > mentioned separately. > > > these 3 items need to be removed, because of > https://git.postgresql.org/cgit/postgresql.git/commit/?id=8aee330af55d8a759b2b73f5a771d9d34a7b887f > > >> Add stratnum GiST support function (Paul A. Jungwirth) > > >> Allow foreign keys to reference WITHOUT OVERLAPS primary keys (Paul A. > >> Jungwirth) > >> The keyword PERIOD is used for this purpose. > > >> Allow PRIMARY KEY and UNIQUE constraints to use WITHOUT OVERLAPS for > >> non-overlapping exclusion constraints (Paul A. Jungwirth) > > > [1] > https://git.postgresql.org/cgit/postgresql.git/commit/?id=519fc1bd9e9d7b408903e44f55f83f6db30742b7 > [2] > https://www.postgresql.org/message-id/CAEZATCUJ0xjyQUL7SHKxJ5a%2BDm5pjoq-WO3NtkDLi6c76rh58w%40mail.gmail.com Agreed, I have applied the attached patch to make the release notes current. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you. diff --git a/doc/src/sgml/release-17.sgml b/doc/src/sgml/release-17.sgml index 9c511848943..0bc1c9a14ad 100644 --- a/doc/src/sgml/release-17.sgml +++ b/doc/src/sgml/release-17.sgml @@ -6,7 +6,7 @@ Release date: - 2024-??-??, AS OF 2024-05-14 + 2024-??-??, AS OF 2024-05-18 @@ -480,17 +480,6 @@ Author: Tomas Vondra Allow BRIN indexes to be created using parallel workers (Tomas Vondra, Matthias van de Meent) - - - - - - -Add stratnum GiST support function (Paul A. Jungwirth) - @@ -1467,34 +1456,6 @@ Add DEFAULT setting for ALTER TABLE .. SET ACCESS METHOD (Michael Paquier) - - - - -Allow foreign keys to reference WITHOUT OVERLAPS primary keys (Paul A. Jungwirth) - - - -The keyword PERIOD is used for this purpose. - - - - - - - -Allow PRIMARY KEY and UNIQUE constraints to use WITHOUT OVERLAPS for non-overlapping exclusion constraints (Paul A. Jungwirth) - - -
Re: First draft of PG 17 release notes
On Thu, May 16, 2024 at 11:50:20AM -0400, Andrew Dunstan wrote: > > Maybe "Introduce an incremental JSON parser" would have been a better > > > headline. > > Well, this gets into a level of detail that is beyond the average > > reader. I think at that level people will need to read the git logs or > > review the code. Do we use it for anything yet? > > > Yes, certainly, it's used in handling backup manifests. Without it we can't > handle huge manifests. See commits ea7b4e9a2a and 222e11a10a. > > Other uses are in the works. Okay, added in the attached applied patch. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you. diff --git a/doc/src/sgml/release-17.sgml b/doc/src/sgml/release-17.sgml index 428cb5c5a2e..9c511848943 100644 --- a/doc/src/sgml/release-17.sgml +++ b/doc/src/sgml/release-17.sgml @@ -2451,6 +2451,17 @@ User-defined data type receive functions will no longer receive their data null- + + + + +Add incremental JSON parser for use with huge JSON documents (Andrew Dunstan) + + +
Re: First draft of PG 17 release notes
On Thu, May 16, 2024 at 04:29:38PM +0800, jian he wrote:
> On Thu, May 9, 2024 at 12:04 PM Bruce Momjian wrote:
> >
> > I have committed the first draft of the PG 17 release notes; you can
> > see the results here:
> >
> > https://momjian.us/pgsql_docs/release-17.html
> >
>
> >> Add jsonpath methods to convert JSON values to different data types
> >> (Jeevan Chalke)
> >> The jsonpath methods are .bigint(), .boolean(), .date(),
> >> .decimal([precision [, scale]]), .integer(), .number(), .string(),
> >> .time(), .time_tz(), .timestamp(), and .timestamp_tz().
>
> I think it's slightly incorrect.
>
> for example:
> select jsonb_path_query('"2023-08-15"', '$.date()');
> I think it does is trying to validate json value "2023-08-15" can be a
> date value, if so, print json string out, else error out.
>
>
> "convert JSON values to different data types"
> meaning that we are converting json values to another data type, date?
I see your point. I have reworded it to be:
Add jsonpath methods to convert JSON values to other JSON data
types (Jeevan Chalke)
Does that help? I think your example is causing confusion because once
JSON values enter the SQL data type space, they are strings.
--
Bruce Momjian https://momjian.us
EDB https://enterprisedb.com
Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Thu, May 16, 2024 at 09:09:11AM -0400, Melanie Plageman wrote: > On Wed, May 15, 2024 at 11:48 PM Andres Freund wrote: > > > > On 2024-05-15 10:38:20 +0200, Alvaro Herrera wrote: > > > I disagree with this. IMO the impact of the Sawada/Naylor change is > > > likely to be enormous for people with large tables and large numbers of > > > tuples to clean up (I know we've had a number of customers in this > > > situation, I can't imagine any Postgres service provider that doesn't). > > > The fact that maintenance_work_mem is no longer capped at 1GB is very > > > important and I think we should mention that explicitly in the release > > > notes, as setting it higher could make a big difference in vacuum run > > > times. > > > > +many. > > > > We're having this debate every release. I think the ongoing reticence to > > note > > performance improvements in the release notes is hurting Postgres. > > > > For one, performance improvements are one of the prime reason users > > upgrade. Without them being noted anywhere more dense than the commit log, > > it's very hard to figure out what improved for users. A halfway widely > > applicable performance improvement is far more impactful than many of the > > feature changes we do list in the release notes. > > The practical reason this matters to users is that they want to change > their configuration or expectations in response to performance > improvements. > > And also, as Jelte mentions upthread, describing performance > improvements made each release in Postgres makes it clear that we are > consistently improving it. > > > For another, it's also very frustrating for developers that focus on > > performance. The reticence to note their work, while noting other, far > > smaller, things in the release notes, pretty much tells us that our work > > isn't > > valued. > > +many Please see the email I just posted. There are three goals we have to adjust for: 1. short release notes so they are readable 2. giving people credit for performance improvements 3. showing people Postgres cares about performance I would like to achieve 2 & 3 without harming #1. My experience is if I am reading a long document, and I get to a section where I start to wonder, "Why should I care about this?", I start to skim the rest of the document. I am particularly critical if I start to wonder, "Why does the author _think_ I should care about this?" becasue it feels like the author is writing for him/herself and not the audience. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Wed, May 15, 2024 at 08:48:02PM -0700, Andres Freund wrote: > Hi, > > On 2024-05-15 10:38:20 +0200, Alvaro Herrera wrote: > > I disagree with this. IMO the impact of the Sawada/Naylor change is > > likely to be enormous for people with large tables and large numbers of > > tuples to clean up (I know we've had a number of customers in this > > situation, I can't imagine any Postgres service provider that doesn't). > > The fact that maintenance_work_mem is no longer capped at 1GB is very > > important and I think we should mention that explicitly in the release > > notes, as setting it higher could make a big difference in vacuum run > > times. > > +many. > > We're having this debate every release. I think the ongoing reticence to note > performance improvements in the release notes is hurting Postgres. > > For one, performance improvements are one of the prime reason users > upgrade. Without them being noted anywhere more dense than the commit log, > it's very hard to figure out what improved for users. A halfway widely > applicable performance improvement is far more impactful than many of the > feature changes we do list in the release notes. I agree the impact of performance improvements are often greater than the average release note item. However, if people expect Postgres to be faster, is it important for them to know _why_ it is faster? If we add a new flag to a command, people will want to know about it so they can make use of it, or if there is a performance improvement that allows new workloads, they will want to know about that too so they can consider those workloads. On the flip side, a performance improvement that makes everything 10% faster has little behavioral change for users, and in fact I think we get ~6% faster in every major release. > For another, it's also very frustrating for developers that focus on > performance. The reticence to note their work, while noting other, far > smaller, things in the release notes, pretty much tells us that our work isn't > valued. Yes, but are we willing to add text that every user will have to read just for this purpose? One think we _could_ do is to create a generic performance release note item saying performance has been improved in the following areas, with no more details, but we can list the authors on the item. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Thu, May 16, 2024 at 03:35:17PM +1200, David Rowley wrote: > On Thu, 16 May 2024 at 14:48, Bruce Momjian wrote: > > > > On Wed, May 15, 2024 at 09:13:14AM -0400, Melanie Plageman wrote: > > > Also +1 on the Sawada/Naylor change being on the highlight section of > > > the release (as David suggested upthread). > > > > Agreed, I went with the attached applied patch. > > +Allow vacuum to more efficiently store tuple references and remove > its memory limit (Masahiko Sawada, John Naylor) > + > > I don't want it to seem like I'm splitting hairs, but I'd drop the " > and remove its memory limit" > > + > +Specifically, maintenance_work_mem and autovacuum_work_mem can now be > configured to use more than one gigabyte of memory. WAL traffic > caused by vacuum is also more compact. > > I'd say the first sentence above should be written as: > > "Additionally, vacuum no longer silently imposes a 1GB tuple reference > limit even when maintenance_work_mem or autovacuum_work_mem are set to > higher values" > > It's not "Specifically" as the "more efficiently store tuple > references" isn't the same thing as removing the 1GB cap. Also, there > was never a restriction in configuring maintenance_work_mem or > autovacuum_work_mem to values higher than 1GB. The restriction was > that vacuum was unable to utilize anything more than that. Slightly adjusted wording patch attached and applied. My deep apologies for the delay in addressing this. I should have done it sooner. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you. diff --git a/doc/src/sgml/release-17.sgml b/doc/src/sgml/release-17.sgml index ed1af17bb0a..d7fd87f3d5c 100644 --- a/doc/src/sgml/release-17.sgml +++ b/doc/src/sgml/release-17.sgml @@ -530,11 +530,12 @@ Author: Heikki Linnakangas -Allow vacuum to more efficiently store tuple references and remove its memory limit (Masahiko Sawada, John Naylor) +Allow vacuum to more efficiently store tuple references (Masahiko Sawada, John Naylor) -Specifically, maintenance_work_mem and autovacuum_work_mem can now be configured to use more than one gigabyte of memory. WAL traffic caused by vacuum is also more compact. +Additionally, vacuum is no longer silently limited to one gigabyte of memory when maintenance_work_mem and autovacuum_work_mem are higher. WAL traffic caused by +vacuum is also more compact.
Re: Fix log_line_prefix to display the transaction id (%x) for statements not in a transaction block
On Thu, May 16, 2024 at 08:37:43AM -0400, Robert Haas wrote: > On Thu, May 16, 2024 at 6:01 AM Quan Zongliang wrote: > > I thought about writing statement log when xid assigned. But it's seemed > > too complicated. > > I'm inclined to keep it for a while. Until we find a good way or give > > up. It's a reasonable request, after all. > > I don't think it's reasonable at all. We can't log the XID before it's > assigned, and we can't log the statement after the XID is assigned > without completely changing how the parameter works. I have removed the TODO item. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: More links on release-17.html
On Wed, May 15, 2024 at 04:50:47PM -0300, Marcos Pegoraro wrote: > While seeing changes and new features of > https://www.postgresql.org/docs/devel/release-17.html > I see that there are too little links to other DOC pages, which would be > useful. > > There are links to > "logical-replication", "sql-merge", "plpgsql", "libpq" and "pgstatstatements" > > But no one link is available to > COPY "ON_ERROR ignore", pg_dump, JSON_TABLE(), xmltext(), pg_basetype() , and > a > lot of other important features. So, wouldn't it be good to have their own > links, so the reader doesn't need to manually search for that feature ? Yes, it would be nice to have them. I will be looking for them in the coming weeks. I usually choose the closest link. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: cataloguing NOT NULL constraints
On Wed, May 15, 2024 at 09:50:36AM +0200, Álvaro Herrera wrote: > On 2024-May-14, Bruce Momjian wrote: > > > Turns out these commits generated a single release note item, which I > > have now removed with the attached committed patch. > > Hmm, but the commits about not-null constraints for domains were not > reverted, only the ones for constraints on relations. I think the > release notes don't properly address the ones on domains. I think it's > at least these two commits: > > > -Author: Peter Eisentraut > > -2024-03-20 [e5da0fe3c] Catalog domain not-null constraints > > -Author: Peter Eisentraut > > -2024-04-15 [9895b35cb] Fix ALTER DOMAIN NOT NULL syntax > > It may still be a good idea to make a note about those, at least to > point out that information_schema now lists them. For example, pg11 > release notes had this item Let me explain what I did to adjust the release notes. I took your commit hashes, which were longer than mine, and got the commit subject text from them. I then searched the release notes to see which commit subjects existed in the document. Only the first three did, and the release note item has five commits. The then tested if the last two patches could be reverted, and 'patch' thought they could be, so that confirmed they were not reverted. However, there was no text in the release note item that corresponded to the commits, so I just removed the entire item. What I now think happened is that the last two commits were considered part of the larger NOT NULL change, and not worth mentioning separately, but now that the NOT NULL part is reverted, we might need to mention them. I rarely handle such complex cases so I don't think I was totally correct in my handling. Let's get a reply to Peter Eisentraut's question and we can figure out what to do. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Thu, May 16, 2024 at 10:39:18AM +0800, jian he wrote: > On Thu, May 9, 2024 at 12:04 PM Bruce Momjian wrote: > > > > I have committed the first draft of the PG 17 release notes; you can > > see the results here: > > > > https://momjian.us/pgsql_docs/release-17.html > > > > >> Add local I/O block read/write timing statistics columns of > >> pg_stat_statement (Nazir Bilal Yavuz) > >> The new columns are "local_blk_read_time" and "local_blk_write_time". > here, "pg_stat_statement" should be "pg_stat_statements"? Agreed. > >> Add optional fourth parameter to pg_stat_statements_reset() to allow for > >> the resetting of only min/max statistics (Andrei Zubkov) > >> This parameter defaults to "false". > here, "parameter", should be "argument"? > > maybe > >> Add optional fourth boolean argument (minmax_only) to > >> pg_stat_statements_reset() to allow for the resetting of only min/max > >> statistics (Andrei Zubkov) > >> This argument defaults to "false". Sure. > > in section: E.1.2. Migration to Version 17 > > >> Rename I/O block read/write timing statistics columns of pg_stat_statement > >> (Nazir Bilal Yavuz) > >> This renames "blk_read_time" to "shared_blk_read_time", and > >> "blk_write_time" to "shared_blk_write_time". > > "pg_stat_statement" should be "pg_stat_statements"? Yes, fixed. > also, we only mentioned, pg_stat_statements some columns name changed > in "E.1.2. Migration to Version 17" > but if you look at the release note pg_stat_statements section, > we added a bunch of columns, which are far more incompatibile than > just colunm name changes. > > not sure we need add these in section "E.1.2. Migration to Version 17" Well, new columns don't cause breakage like renamed columns, which is why I only put renames/removed columns in the migration section. Also, thanks everyone for the release notes feedback. In some cases I made a mistake, and in some cases I misjudged the item. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Wed, May 15, 2024 at 09:13:14AM -0400, Melanie Plageman wrote: > I think this wording and organization makes sense. I hadn't thought of > using "traffic" to describe this, but I like it. > > Also +1 on the Sawada/Naylor change being on the highlight section of > the release (as David suggested upthread). Agreed, I went with the attached applied patch. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you. diff --git a/doc/src/sgml/release-17.sgml b/doc/src/sgml/release-17.sgml index e68c499e0db..7f36a35954d 100644 --- a/doc/src/sgml/release-17.sgml +++ b/doc/src/sgml/release-17.sgml @@ -519,6 +519,25 @@ Allow vacuum to more efficiently remove and freeze tuples (Masahiko Sawada, John + + + + +Allow vacuum to more efficiently store tuple references and remove its memory limit (Masahiko Sawada, John Naylor) + + + +Specifically, maintenance_work_mem and autovacuum_work_mem can now be configured to use more than one gigabyte of memory. WAL traffic caused by vacuum is also more compact. + + +
Re: First draft of PG 17 release notes
On Wed, May 15, 2024 at 10:10:28AM +0900, Masahiko Sawada wrote: > > I looked at that item and I don't have a generic "make logical > > replication apply faster" item to merge it into, and many > > subtransactions seemed like enough of an edge-case that I didn't think > > mentioning it make sense. Can you see a good place to add it? > > I think that since many subtransactions cases are no longer becoming > edge-cases these days, we needed to improve that and it might be > helpful for users to mention it. How about the following item for > example? > > Improve logical decoding performance in cases where there are many > subtransactions. Okay, item added in the attached applied patch. > > > Finally, should we mention the following commit in the release note? > > > It's not a user-visible change but added a new regression test module. > > > > > > - Add tests for XID wraparound (e255b646a) > > > > I don't normally add testing infrastructure changes unless they are > > major. > > I've seen we had such item, for example in PG14 release note: > > Add a test module for the regular expression package (Tom Lane) > > But if our policy has already changed, I'm okay with not mentioning > the xid_wraparound test in the PG17 release note. Uh, that PG 14 test suite was huge and flushed out a lot of bugs, not only in our regex code but I think in the TCL/Henry Spencer regex library we inherited. We add 10-40 tests every year, and how many do I mention in the release notes? You had to go back to PG 14 to find one. We have not changed our release note "test item" criteria --- I only mention tests that are significant to our userbase. I think that test suite was significant to anyone using the TCL/Henry Spencer regex library. If you want your test mentioned, you have to explain why it is useful for users to know about it, or the value it brings them. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you. diff --git a/doc/src/sgml/release-17.sgml b/doc/src/sgml/release-17.sgml index 510810b53dc..e68c499e0db 100644 --- a/doc/src/sgml/release-17.sgml +++ b/doc/src/sgml/release-17.sgml @@ -1150,6 +1150,17 @@ Previously only btree indexes could be used for this purpose. + + + + +Improve logical decoding performance in cases where there are many subtransactions (Masahiko Sawada) + + +
Re: First draft of PG 17 release notes
On Wed, May 15, 2024 at 02:03:32PM +1200, David Rowley wrote: > On Wed, 15 May 2024 at 13:00, Bruce Momjian wrote: > > > > On Tue, May 14, 2024 at 03:39:26PM -0400, Melanie Plageman wrote: > > > "Reduce system calls by automatically merging reads up to > > > io_combine_limit" > > > > Uh, as I understand it, the reduced number of system calls is not the > > value of the feature, but rather the ability to request a larger block > > from the I/O subsystem. Without it, you have to make a request and wait > > for each request to finish. I am open to new wording, but I am not sure > > your new wording is accurate. > > I think you have the cause and effect backwards. There's no advantage > to reading 128KB if you only need 8KB. It's the fact that doing > *larger* reads allows *fewer* reads that allows it to be more > efficient. There are also the efficiency gains from fadvise > POSIX_FADV_WILLNEED. I'm unsure how to jam that into a short sentence. > Maybe; "Optimize reading of tables by allowing pages to be prefetched > and read in chunks up to io_combine_limit", or a bit more buzzy; > "Optimize reading of tables by allowing pages to be prefetched and > performing vectored reads in chunks up to io_combine_limit". Yes, my point is that it is not the number of system calls or system call overhead that is the advantage of this patch, but the ability to request more of the I/O system in one call, which is not the same as system calls. I can use your wording, but how much prefetching to we enable now? -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Tue, May 14, 2024 at 10:22:35AM +0800, Tender Wang wrote: > > > jian he 于2024年5月9日周四 18:00写道： > > On Thu, May 9, 2024 at 12:04 PM Bruce Momjian wrote: > > > > I have committed the first draft of the PG 17 release notes; you can > > see the results here: > > > > https://momjian.us/pgsql_docs/release-17.html > > > > another potential incompatibilities issue: > ALTER TABLE DROP PRIMARY KEY > > see: > https://www.postgresql.org/message-id/ > 202404181849.6frtmajobe27%40alvherre.pgsql > > > > Since Alvaro has reverted all changes to not-null constraints, so will not > have > potential incompatibilities issue. Agreed. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: cataloguing NOT NULL constraints
On Mon, May 13, 2024 at 09:00:28AM -0400, Robert Haas wrote: > > Specifically, the problem is that I mentioned that we could restrict the > > NOT NULL NO INHERIT addition in pg_dump for primary keys to occur only > > in pg_upgrade; but it turns this is not correct. In normal > > dump/restore, there's an additional table scan to check for nulls when > > the constraints is not there, so the PK creation would become measurably > > slower. (In a table with a million single-int rows, PK creation goes > > from 2000ms to 2300ms due to the second scan to check for nulls). > > I have a feeling that any theory of the form "X only needs to happen > during pg_upgrade" is likely to be wrong. pg_upgrade isn't really > doing anything especially unusual: just creating some objects and > loading data. Those things can also be done at other times, so > whatever is needed during pg_upgrade is also likely to be needed at > other times. Maybe that's not sound reasoning for some reason or > other, but that's my intuition. I assume Alvaro is saying that pg_upgrade has only a single session, which is unique and might make things easier for him. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: cataloguing NOT NULL constraints
On Sun, May 12, 2024 at 04:56:09PM +0200, Álvaro Herrera wrote: > On 2024-May-11, Alvaro Herrera wrote: > > > I have found two more problems that [] require some more work to fix, > > so I've decided to cut my losses now and revert the whole. > > Here's the revert patch, which I intend to push early tomorrow. > > Commits reverted are: > 21ac38f498b33f0231842238b83847ec63dfe07b > d45597f72fe53a53f6271d5ba4e7acf8fc9308a1 > 13daa33fa5a6d340f9be280db14e7b07ed11f92e > 0cd711271d42b0888d36f8eda50e1092c2fed4b3 > d72d32f52d26c9588256de90b9bc54fe312cee60 > d9f686a72ee91f6773e5d2bc52994db8d7157a8e > c3709100be73ad5af7ff536476d4d713bca41b1a > 3af7217942722369a6eb7629e0fb1cbbef889a9b > b0f7dd915bca6243f3daf52a81b8d0682a38ee3b > ac22a9545ca906e70a819b54e76de38817c93aaf > d0ec2ddbe088f6da35444fad688a62eae4fbd840 > 9b581c53418666205938311ef86047aa3c6b741f > b0e96f311985bceba79825214f8e43f65afa653a > > with some significant conflict fixes (mostly in the last one). Turns out these commits generated a single release note item, which I have now removed with the attached committed patch. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you. diff --git a/doc/src/sgml/release-17.sgml b/doc/src/sgml/release-17.sgml index 9c7c0a0337f..143ee17716d 100644 --- a/doc/src/sgml/release-17.sgml +++ b/doc/src/sgml/release-17.sgml @@ -1486,29 +1486,6 @@ Add event trigger support for REINDEX (Garrett Thornburg, Jian He) - - - - -Allow NOT NULL columns to be specified as column constraints and domains (Alvaro Herrera, Bernd Helmle, Kyotaro Horiguchi, Peter Eisentraut) - - - -Previously NOT NULL could only be specified as a column attribute. - - -
Re: First draft of PG 17 release notes
On Tue, May 14, 2024 at 03:39:26PM -0400, Melanie Plageman wrote: > On Thu, May 9, 2024 at 12:04 AM Bruce Momjian wrote: > > > > I have committed the first draft of the PG 17 release notes; you can > > see the results here: > > > > https://momjian.us/pgsql_docs/release-17.html > > I had two comments: > > > I think the read stream item: > > "Allow the grouping of file system reads with the new system variable > io_combine_limit" > > Might be better if it mentions the effect, like: > > "Reduce system calls by automatically merging reads up to io_combine_limit" Uh, as I understand it, the reduced number of system calls is not the value of the feature, but rather the ability to request a larger block from the I/O subsystem. Without it, you have to make a request and wait for each request to finish. I am open to new wording, but I am not sure your new wording is accurate. > --- > For the vacuum feature: > > "Allow vacuum to more efficiently remove and freeze tuples" > > I think that we need to more clearly point out the implications of the > feature added by Sawada-san (and reviewed by John) in 667e65aac35497. > Vacuum no longer uses a fixed amount of memory for dead tuple TID > storage and it is not preallocated. This affects users as they may > want to change their configuration (and expectations). > > If you make that item more specific to their work, you should also > remove my name, as the work I did on vacuum this release was unrelated > to their work on dead tuple TID storage. > > The work Heikki and I did which culminated in 6dbb490261 mainly has > the impact of improving vacuum's performance (vacuum emits less WAL > and is more efficient). So you could argue for removing it from the > release notes if you are using the requirement that performance > improvements don't go in the release notes. > > However, one of the preliminary commits for this f83d70976 does change > WAL format. There are three WAL records which no longer exist as > separate records. Do users care about this? I don't think users really care about these details, just that it is faster so they will not be surprised if there is a change. I was purposely vague to group multiple commits into the single item. By grouping them together, I got enough impact to warrant listing it. If you split it apart, it is harder to justify mentioning them. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Tue, May 14, 2024 at 02:58:41PM +0100, Pantelis Theodosiou wrote: > On Thu, May 9, 2024 at 5:03 AM Bruce Momjian wrote > > > > > > I welcome feedback. For some reason it was an easier job than usual. > > This looks better if "more case" -> "more cases" : > > Allow query nodes to be run in parallel in more case (Tom Lane) Yes, you are correct, fixed. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Tue, May 14, 2024 at 02:20:24PM +0200, Jelte Fennema-Nio wrote: > On Tue, 14 May 2024 at 02:56, Bruce Momjian wrote: > > > > On Sat, May 11, 2024 at 10:24:39AM -0400, Joe Conway wrote: > > > On 5/11/24 09:57, Jelte Fennema-Nio wrote: > > > > The buffering change improved performance up to ~40% in some of the > > > > benchmarks. The case it improves mostly is COPY of large rows and > > > > streaming a base backup. That sounds user-visible enough to me to > > > > warrant an entry imho. > > > > > > +1 > > > > Attached patch applied. > > I think we shouldn't list this under the libpq changes and shouldn't > mention libpq in the description, since this patch changes > src/backend/libpq files instead of src/interfaces/libpq files. I think > it should be in the "General performance" section and describe the > change as something like the below: > > Improve performance when transferring large blocks of data to a client > > PS. I completely understand that this was not clear from the commit message. Okay, I went with your wording. Attached patch applied. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you. diff --git a/doc/src/sgml/release-17.sgml b/doc/src/sgml/release-17.sgml index a3b4a8fb3b9..448f5653bc5 100644 --- a/doc/src/sgml/release-17.sgml +++ b/doc/src/sgml/release-17.sgml @@ -563,6 +563,17 @@ Improve performance of heavily-contended WAL writes (Bharath Rupireddy) + + + + +Improve performance when transferring large blocks of data to a client (Melih Mutlu) + + + - - - -Allow libpq to more efficiently transfer large blocks of data (Melih Mutlu) - - -
Re: First draft of PG 17 release notes
On Tue, May 14, 2024 at 01:34:56PM +0300, Elena Indrupskaya wrote: > Being a technical writer, I attached a small patch that fixes minor language > stuff. You are absolutely correct. Patch applied, thanks. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Sat, May 11, 2024 at 03:32:55PM -0400, Andrew Dunstan wrote: > > On 2024-05-09 Th 00:03, Bruce Momjian wrote: > > I have committed the first draft of the PG 17 release notes; you can > > see the results here: > > > > https://momjian.us/pgsql_docs/release-17.html > > > > It will be improved until the final release. The item count is 188, > > which is similar to recent releases: > > > > release-10: 189 > > release-11: 170 > > release-12: 180 > > release-13: 178 > > release-14: 220 > > release-15: 184 > > release-16: 206 > > release-17: 188 > > > > I welcome feedback. For some reason it was an easier job than usual. > > > I don't like blowing my own horn but I feel commit 3311ea86ed "Introduce a > non-recursive JSON parser" should be in the release notes. This isn't > something that's purely internal, but it could be used by an extension or a > client program to parse JSON documents that are too large to handle with the > existing API. > > Maybe "Introduce an incremental JSON parser" would have been a better > headline. Well, this gets into a level of detail that is beyond the average reader. I think at that level people will need to read the git logs or review the code. Do we use it for anything yet? It could be put in the source code section but I try to only have user-visible stuff there. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Tue, May 14, 2024 at 10:32:14AM +0800, Andy Fan wrote: > Bruce Momjian writes: > > It was unclear from the commit message exactly what user-visible > > optimization this allowed. Do you have details? > > Yes, It allows the query like "SELECT * FROM t1 WHERE t1.a in (SELECT a > FROM t2 WHERE t2.b = t1.b)" be pulled up a semi join, hence more join > methods / join orders are possible. > > > Yes, for example: (subquery-1) UNION ALL (subquery-2) LIMIT n; > > When planning the subquery-1 or subquery-2, limit N should be > considered. As a consequence, maybe hash join should be replaced with > Nested Loop. Before this commits, it is ignored if it is flatten into > appendrel, and the "flatten" happens very often. > > David provided a summary for the both commits in [1]. Okay, attached patch applied. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you. diff --git a/doc/src/sgml/release-17.sgml b/doc/src/sgml/release-17.sgml index 38c14970822..fa0a703629c 100644 --- a/doc/src/sgml/release-17.sgml +++ b/doc/src/sgml/release-17.sgml @@ -356,6 +356,28 @@ Improve optimization of range values when using containment operators @ and + + + + +Allow correlated IN subqueries to be transformed into joins (Andy Fan, Tom Lane) + + + + + + + +Improve optimization of the LIMIT clause on partitioned tables, inheritance parents, and UNION ALL queries (Andy Fan, David Rowley) + + +
Re: First draft of PG 17 release notes
On Sat, May 11, 2024 at 01:27:25PM +0800, Andy Fan wrote: > > Hello Bruce, > > > I have committed the first draft of the PG 17 release notes; you can > > see the results here: > > > > https://momjian.us/pgsql_docs/release-17.html > > Thank you for working on this! > > > I welcome feedback. For some reason it was an easier job than usual. > > Do you think we need to add the following 2 items? > > - 9f133763961e280d8ba692bcad0b061b861e9138 this is an optimizer > transform improvement. It was unclear from the commit message exactly what user-visible optimization this allowed. Do you have details? > - a8a968a8212ee3ef7f22795c834b33d871fac262 this is an optimizer costing > improvement. Does this allow faster UNION ALL with LIMIT, perhaps? -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Sat, May 11, 2024 at 10:24:39AM -0400, Joe Conway wrote: > On 5/11/24 09:57, Jelte Fennema-Nio wrote: > > On Fri, 10 May 2024 at 23:31, Tom Lane wrote: > > > > > > Bruce Momjian writes: > > > > I looked at both of these. In both cases I didn't see why the user > > > > would need to know these changes were made: > > > > > > I agree that the buffering change is not likely interesting, but > > > the fact that you can now control-C out of a psql "\c" command > > > is user-visible. People might have internalized the fact that > > > it didn't work, or created complicated workarounds. > > > > The buffering change improved performance up to ~40% in some of the > > benchmarks. The case it improves mostly is COPY of large rows and > > streaming a base backup. That sounds user-visible enough to me to > > warrant an entry imho. > > +1 Attached patch applied. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you. commit e87e7324555 Author: Bruce Momjian Date: Mon May 13 20:55:13 2024 -0400 doc PG 17 relnotes: add item about libpq large data transfers Reported-by: Jelte Fennema-Nio Discussion: https://postgr.es/m/cageczqtz5auqlel6dald2hu2fxs_losh4kedndj1fwthsb_...@mail.gmail.com Reviewed-by: Joe Conway Backpatch-through: master diff --git a/doc/src/sgml/release-17.sgml b/doc/src/sgml/release-17.sgml index 9dd3954f3c2..38c14970822 100644 --- a/doc/src/sgml/release-17.sgml +++ b/doc/src/sgml/release-17.sgml @@ -1901,6 +1901,17 @@ Add libpq function PQsetChunkedRowsMode to allow retrieval of results in chunks + + + + +Allow libpq to more efficiently transfer large blocks of data (Melih Mutlu) + + +
Re: First draft of PG 17 release notes
On Fri, May 10, 2024 at 05:31:33PM -0400, Tom Lane wrote: > Bruce Momjian writes: > > On Fri, May 10, 2024 at 06:50:54PM +0200, Jelte Fennema-Nio wrote: > >> There are two commits that I think would benefit from being listed > >> (but maybe they are already listed and I somehow missed them, or they > >> are left out on purpose for some reason): > > > I looked at both of these. In both cases I didn't see why the user > > would need to know these changes were made: > > I agree that the buffering change is not likely interesting, but > the fact that you can now control-C out of a psql "\c" command > is user-visible. People might have internalized the fact that > it didn't work, or created complicated workarounds. Agreed, attached patch applied. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you. diff --git a/doc/src/sgml/release-17.sgml b/doc/src/sgml/release-17.sgml index 5a741ff16f1..9dd3954f3c2 100644 --- a/doc/src/sgml/release-17.sgml +++ b/doc/src/sgml/release-17.sgml @@ -1980,6 +1980,17 @@ The parameter is "min_rows". + + + + +Allow psql connections to be canceled with control-C (Tristan Partin) + + +
Re: First draft of PG 17 release notes
On Fri, May 10, 2024 at 05:31:33PM -0400, Tom Lane wrote: > Bruce Momjian writes: > > On Fri, May 10, 2024 at 06:50:54PM +0200, Jelte Fennema-Nio wrote: > >> There are two commits that I think would benefit from being listed > >> (but maybe they are already listed and I somehow missed them, or they > >> are left out on purpose for some reason): > > > I looked at both of these. In both cases I didn't see why the user > > would need to know these changes were made: > > I agree that the buffering change is not likely interesting, but > the fact that you can now control-C out of a psql "\c" command > is user-visible. People might have internalized the fact that > it didn't work, or created complicated workarounds. It was not clear to me what the user-visible behavior was with the SIGINT control. Yes, based on your details, it should be mentioned. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Fri, May 10, 2024 at 06:50:54PM +0200, Jelte Fennema-Nio wrote: > On Thu, 9 May 2024 at 06:04, Bruce Momjian wrote: > > > > I have committed the first draft of the PG 17 release notes; you can > > see the results here: > > > > https://momjian.us/pgsql_docs/release-17.html > > Great work! > > There are two commits that I think would benefit from being listed > (but maybe they are already listed and I somehow missed them, or they > are left out on purpose for some reason): I looked at both of these. In both cases I didn't see why the user would need to know these changes were made: --- > - c4ab7da60617f020e8d75b1584d0754005d71830 commit c4ab7da6061 Author: David Rowley Date: Sun Apr 7 21:20:18 2024 +1200 Avoid needless large memcpys in libpq socket writing Until now, when calling pq_putmessage to write new data to a libpq socket, all writes are copied into a buffer and that buffer gets flushed when full to avoid having to perform small writes to the socket. There are cases where we must write large amounts of data to the socket, sometimes larger than the size of the buffer. In this case, it's wasteful to memcpy this data into the buffer and flush it out, instead, we can send it directly from the memory location that the data is already stored in. Here we adjust internal_putbytes() so that after having just flushed the buffer to the socket, if the remaining bytes to send is as big or bigger than the buffer size, we just send directly rather than needlessly copying into the PqSendBuffer buffer first. Examples of operations that write large amounts of data in one message are; outputting large tuples with SELECT or COPY TO STDOUT and pg_basebackup. Author: Melih Mutlu Reviewed-by: Heikki Linnakangas Reviewed-by: Jelte Fennema-Nio Reviewed-by: David Rowley Reviewed-by: Ranier Vilela Reviewed-by: Andres Freund Discussion: https://postgr.es/m/cagpvpcr15nosj0f6xe-c2h477zfr88q12e6wjeoezc8zykt...@mail.gmail.com > - cafe1056558fe07cdc52b95205588fcd80870362 commit cafe1056558 Author: Robert Haas Date: Tue Apr 2 10:26:10 2024 -0400 Allow SIGINT to cancel psql database reconnections. After installing the SIGINT handler in psql, SIGINT can no longer cancel database reconnections. For instance, if the user starts a reconnection and then needs to do some form of interaction (ie psql is polling), there is no way to cancel the reconnection process currently. Use PQconnectStartParams() in order to insert a cancel_pressed check into the polling loop. Tristan Partin, reviewed by Gurjeet Singh, Heikki Linnakangas, Jelte Fennema-Nio, and me. Discussion: http://postgr.es/m/d08wwcpvhkhn.3qelikzj2d...@neon.tech -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: Augmenting the deadlock message with application_name
On Fri, May 10, 2024 at 08:10:58PM +, Karoline Pauls wrote: > On Friday, 10 May 2024 at 20:17, Bruce Momjian > wrote: > > > > log_line_prefix supports application name --- why would you not use > > that? > > > > log_line_prefix is effective in the server log. This change is mostly > about improving the message sent back to the client. While the server > log is also changed to reflect the client message, it doesn't need to > be. I was hoping client_min_messages would show the application name, but it doesn't but your bigger point is below. > Additionally, with `%a` added to log_line_prefix, the server log > would only contain the application name of the client affected by the > deadlock, not the application names of all other clients involved in > it. Yeah, getting the application names of the pids reported in the log requires looking backward in the logs to find out what the most recent log line was for the pids involved. Frankly, I think it would be more useful to show the session_id in the deadlock so you could then use that to look back to any details you want in the logs, not only the application name. > Anyway, the server log is not the important part here. The crucial > UX feature is the client getting application names back, so browsing > through server logs can be avoided. Well, we don't want to report too much information because it gets confusing. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Fri, May 10, 2024 at 06:29:11PM +0200, Daniel Verite wrote: > Bruce Momjian wrote: > > > have committed the first draft of the PG 17 release notes; you can > > see the results here: > > > > https://momjian.us/pgsql_docs/release-17.html > > In the psql items, I'd suggest mentioning > > https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=90f5178 > > For the short description, maybe something like that: > > - Improve FETCH_COUNT to work with all queries (Daniel Vérité) > Previously, results would be fetched in chunks only for queries > that start with the SELECT keyword. Agreed, patch attached and applied. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you. diff --git a/doc/src/sgml/release-17.sgml b/doc/src/sgml/release-17.sgml index e4b34d827d1..08238be9cb7 100644 --- a/doc/src/sgml/release-17.sgml +++ b/doc/src/sgml/release-17.sgml @@ -1893,8 +1893,6 @@ This is similar to PQpipelineSync() but it does not flush to the server unless t @@ -1937,6 +1935,17 @@ This is enabled with the client-side option sslnegotation=direct, requires ALPN, + + + + +Allow FETCH_COUNT to work with non-SELECT queries (Daniel Vérité) + + +
Re: Augmenting the deadlock message with application_name
On Thu, May 9, 2024 at 11:44:03PM +, Karoline Pauls wrote: > As we know, the deadlock error message isn't the most friendly one. All the > client gets back is process PIDs, transaction IDs, and lock types. You have to > check the server log to retrieve lock details. This is tedious. > > In one of my apps I even added a deadlock exception handler on the app side to > query pg_stat_activity for processes involved in the deadlock and include > their > application names and queries in the exception message. It is a little racy > but > works well enough. > > Ideally I'd like to see that data coming from Postgres upon detecting the > deadlock. That's why I made this small change. > > The change makes the deadlock error look as follows - the new element is the > application name or "" in its place if the activity > user doesn't match the current user (and the current use isn't a superuser): > > postgres=*> SELECT * FROM q WHERE id = 2 FOR UPDATE; > ERROR: deadlock detected > DETAIL: Process 194520 (application_name: ) waits for > ShareLock on transaction 776; blocked by process 194521. > Process 194521 (application_name: woof) waits for ShareLock on transaction > 775; > blocked by process 194520. > HINT: See server log for query details. > CONTEXT: while locking tuple (0,2) in relation "q" log_line_prefix supports application name --- why would you not use that? -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Fri, May 10, 2024 at 01:54:30PM +0530, Bharath Rupireddy wrote: > On Thu, May 9, 2024 at 9:34 AM Bruce Momjian wrote: > > > > I have committed the first draft of the PG 17 release notes; you can > > see the results here: > > > > https://momjian.us/pgsql_docs/release-17.html > > > > It will be improved until the final release. The item count is 188, > > which is similar to recent releases: > > > > release-10: 189 > > release-11: 170 > > release-12: 180 > > release-13: 178 > > release-14: 220 > > release-15: 184 > > release-16: 206 > > release-17: 188 > > > > I welcome feedback. For some reason it was an easier job than usual. > > Thanks a lot for this work Bruce! It looks like commit > https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=91f2cae7a4e664e9c0472b364c7db29d755ab151 > is missing from daft release notes. Just curious to know if it's > intentional or a miss out. I did not mention it because the commit didn't mention any performance benefit and it seemed more like an internal change than something people needed to know about. I could reword and merge it into this item, if you think I should: Improve performance of heavily-contended WAL writes (Bharath Rupireddy) -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Thu, May 9, 2024 at 08:40:00PM +0200, Álvaro Herrera wrote: > On 2024-May-09, Bruce Momjian wrote: > > > However, I don't see it mentioned as a release note item in the commit > > message or mentioned in our docs. I suppose the release note text would > > be: > > > > Removing a PRIMARY KEY will remove the NOT NULL column specification > > > > Previously the NOT NULL specification would be retained. > > > > Do we have agreement that we want this release note item? > > Yes. Maybe we want some others too (especially regarding inheritance, > but also regarding the way we handle the constraints internally), and > maybe in this one we want different wording. How about something like > this: > > Removing a primary key constraint may change the nullability > characteristic of the columns that the primary key covered. > > If explicit not-null constraints exist on the same column, then they > continue to be /known not nullable/; otherwise they become /possibly > nullable/. > > This is largely based on the SQL standard's language of a column > descriptor having a "nullability characteristic", which for columns with > not-null or primary key constraints is "known not null". I don't think > we use those terms anywhere. I hope this isn't too confusing. Yes, it was confusing, partly because it is using wording we don't use, and partly because it is talking about what can go into the column, rather than the visible column restriction NOT NULL. I also think "may" is too imprecise. How about: Removing a primary key will remove a column's NOT NULL constraint if the constraint was added by the primary key Previously such NOT NULL constraints would remain after a primary key was removed. A column-level NOT NULL constraint would not be emoved. Here is the PG 16 output: CREATE TABLE test ( x INT CONSTRAINT test_pkey PRIMARY KEY ); Table "public.test" Column | Type | Collation | Nullable | Default +-+---+--+- x | integer | | not null | Indexes: "test_pkey" PRIMARY KEY, btree (x) CREATE TABLE test_with_not_null (x INT NOT NULL CONSTRAINT test_pkey_with_not_null PRIMARY KEY); Table "public.test_with_not_null" Column | Type | Collation | Nullable | Default +-+---+--+- x | integer | | not null | Indexes: "test_pkey_with_not_null" PRIMARY KEY, btree (x) ALTER TABLE test DROP CONSTRAINT test_pkey; Table "public.test" Column | Type | Collation | Nullable | Default +-+---+--+- --> x | integer | | not null | ALTER TABLE test_with_not_null DROP CONSTRAINT test_pkey_with_not_null; Table "public.test_with_not_null" Column | Type | Collation | Nullable | Default +-+---+--+- --> x | integer | | not null | Here is the output in PG 17: CREATE TABLE test ( x INT CONSTRAINT test_pkey PRIMARY KEY ); Table "public.test" Column | Type | Collation | Nullable | Default +-+---+--+- x | integer | | not null | Indexes: "test_pkey" PRIMARY KEY, btree (x) CREATE TABLE test_with_not_null (x INT NOT NULL CONSTRAINT test_pkey_with_not_null PRIMARY KEY); Table "public.test_with_not_null" Column | Type | Collation | Nullable | Default +-+---+--+- x | integer | | not null | Indexes: "test_pkey_with_not_null" PRIMARY KEY, btree (x) ALTER TABLE test DROP CONSTRAINT test_pkey; Table "public.test" Column | Type | Collation | Nullable | Default +-+---+--+- --> x | integer | | | ALTER TABLE test_with_not_null DROP CONSTRAINT test_pkey_with_not_null; Table "public.test_with_not_null" Column | Type | Collation | Nullable | Default +-+---+--+- --> x | integer | | not null | Notice that the table without a _column_ NOT NULL removes the NOT NULL designation after removing the primary key only in PG 17. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Fri, May 10, 2024 at 08:05:43AM +1200, Thomas Munro wrote: > On Thu, May 9, 2024 at 4:04 PM Bruce Momjian wrote: > > I welcome feedback. For some reason it was an easier job than usual. > > > 2024-01-25 [820b5af73] jit: Require at least LLVM 10. > > > Require LLVM version 10 or later (Peter Eisentraut) > > Peter reviewed, I authored, and I think you intend to list authors in > parentheses. Yes, my mistake, fixed. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: PERIOD foreign key feature
On Wed, May 8, 2024 at 08:47:45PM -0700, Paul Jungwirth wrote: > On 5/8/24 07:44, Bruce Momjian wrote: > > On Wed, May 8, 2024 at 02:29:34PM +0200, Peter Eisentraut wrote: > > > > Yes, David is correct here on all points. I like his suggestion to > > > > clarify the language here also. If you need a patch from me let me know, > > > > but I assume it's something a committer can just make happen? > > > > > > In principle yes, but it's also very helpful if someone produces an actual > > > patch file, with complete commit message, credits, mailing list link, etc. > > > > I am ready to do the work, but waited a day for Peter to reply, since he > > was the author of the text. > > Here is a patch for this. Thanks, patch applied. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Thu, May 9, 2024 at 12:10:11PM -0400, Andrew Dunstan wrote:
>
> On 2024-05-09 Th 00:03, Bruce Momjian wrote:
>
> I have committed the first draft of the PG 17 release notes; you can
> see the results here:
>
> https://momjian.us/pgsql_docs/release-17.html
>
> It will be improved until the final release. The item count is 188,
> which is similar to recent releases:
>
> release-10: 189
> release-11: 170
> release-12: 180
> release-13: 178
> release-14: 220
> release-15: 184
> release-16: 206
> release-17: 188
>
> I welcome feedback. For some reason it was an easier job than usual.
>
>
> • Remove the ability to build Postgres with Visual Studio (Michael Paquier)
>
> Meson is now the only available Windows build method.
>
>
> This is a category mistake. What was removed was the special code we had for
> building with VS, but not the ability to build with VS. You can build with VS
> using meson (see for example drongo on the buildfarm)
Wow, okay, I am not surprised I was confused. New text is:
Remove the Microsoft Visual Studio Studio-specific Postgres build
option (Michael Paquier)
Meson is now the only method for Visual Studio builds.
Remove the Microsoft Visual Studio Studio-specific Postgres build
option (Michael Paquier)
Meson is now the only method for Visual Studio builds.
--
Bruce Momjian https://momjian.us
EDB https://enterprisedb.com
Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Thu, May 9, 2024 at 11:26:44PM +0800, jian he wrote: > On Thu, May 9, 2024 at 11:12 PM Bruce Momjian wrote: > > > > On Thu, May 9, 2024 at 07:49:55PM +0800, jian he wrote: > > > On Thu, May 9, 2024 at 6:53 PM jian he > > > wrote: > > > > > > > > On Thu, May 9, 2024 at 12:04 PM Bruce Momjian wrote: > > > > > I have committed the first draft of the PG 17 release notes; you can > > > > > see the results here: > > > > > > > > > > https://momjian.us/pgsql_docs/release-17.html > > > > > E.1.3.1.5. Privileges > Add per-table GRANT permission MAINTAIN to control maintenance > operations (Nathan Bossart) > > The operations are VACUUM, ANALYZE, REINDEX, REFRESH MATERIALIZE VIEW, > CLUSTER, and LOCK TABLE. > > Add user-grantable role pg_maintain to control maintenance operations > (Nathan Bossart) > > The operations are VACUUM, ANALYZE, REINDEX, REFRESH MATERIALIZE VIEW, > CLUSTER, and LOCK TABLE. > > Allow roles with pg_monitor privileges to execute pg_current_logfile() > (Pavlo Golub, Nathan Bossart) > --- > should be "REFRESH MATERIALIZED VIEW"? Yes, fixed. > also > "Allow roles with pg_monitor privileges to execute > pg_current_logfile() (Pavlo Golub, Nathan Bossart)" > "pg_monitor" is a predefined role, so technically, "with pg_monitor > privileges" is not correct? Good point, new text: Allow roles with pg_monitor membership to execute pg_current_logfile() (Pavlo Golub, Nathan Bossart) > -- > Add function XMLText() to convert text to a single XML text node (Jim Jones) > > XMLText() > should be > xmltext() Right, fixed. > -- > Add function to_regtypemod() to return the typemod of a string (David > Wheeler, Erik Wienhold) > I think this description does not mean the same thing as the doc[1] Yes, I see your point. I changed the text to: Add function to_regtypemod() to return the type modifier of a type specification (David Wheeler, Erik Wienhold) > [1] > https://www.postgresql.org/docs/devel/functions-info.html#FUNCTIONS-INFO-CATALOG > -- > > Allow GROUP BY columns to be internally ordered to match ORDER BY > (Andrei Lepikhov, Teodor Sigaev) > This can be disabled using server variable enable_group_by_reordering. > > Probably > `This can be disabled by setting the server variable > enable_group_by_reordering to false`. Uh, I usually don't go into that detail. There will be a link to the variable in about a month so users can look up its behavior. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Thu, May 9, 2024 at 07:49:55PM +0800, jian he wrote: > On Thu, May 9, 2024 at 6:53 PM jian he wrote: > > > > On Thu, May 9, 2024 at 12:04 PM Bruce Momjian wrote: > > > I have committed the first draft of the PG 17 release notes; you can > > > see the results here: > > > > > > https://momjian.us/pgsql_docs/release-17.html > > < Add columns to pg_stats to report range histogram information (Egor > Rogov, Soumyadeep Chakraborty) > I think this applies to range type and multi range type, "range > histogram information" seems not very clear to me. > So maybe: > < Add columns to pg_stats to report range-type histogram information > (Egor Rogov, Soumyadeep Chakraborty) Yes, good point, done. > Display length and bounds histograms in pg_stats Uh, isn't that assumed? Is this a detail worth mentioning? > < Add new COPY option "ON_ERROR ignore" to discard error rows (Damir > Belyalov, Atsushi Torikoshi, Alex Shulgin, Jian He, Jian He, Yugo > Nagata) > duplicate name. Fixed. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Thu, May 9, 2024 at 06:57:01PM +0800, jian he wrote: > > << > > Allow ALTER OPERATOR to set more optimization attributes (Tommy Pavlicek) > > This is useful for extensions. > > << > > sorry, I mean > << > Allow the creation of hash indexes on ltree columns (Tommy Pavlicek) > This also enables hash join and hash aggregation on ltree columns. > << > > better description would be: > << > Add hash support functions and hash opclass for contrib/ltree (Tommy Pavlicek) > This also enables hash join and hash aggregation on ltree columns. > << Yes, please see my previous email where I am asking why being more specific is worse. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Thu, May 9, 2024 at 06:53:30PM +0800, jian he wrote: > On Thu, May 9, 2024 at 12:04 PM Bruce Momjian wrote: > > I have committed the first draft of the PG 17 release notes; you can > > see the results here: > > > > https://momjian.us/pgsql_docs/release-17.html > > > > * Add function pg_buffercache_evict() to allow shared buffer eviction > (Palak Chaturvedi, Thomas Munro) > * This is useful for testing. > > this should put it on the section > < E.1.3.11. Additional Modules > ? Oh, it is in the pg_buffercache module --- I should have realized that from the name, fixed. > Then I found out official release notes don't have attributes, > so it doesn't matter? Uh, what are sections? Did previous release notes have it? > I think this commit title "Add hash support functions and hash opclass > for contrib/ltree." > from [1] is more descriptive. Uh, I don't think people know what hash support functions are, but they know what hash indexes are, and maybe hash joins and hash aggregates. Why do you consider the commit text better? > i am not 100% sure of the meaning of "This is useful for extensions." The commit says: commit 2b5154beab7 Author: Tom Lane Date: Fri Oct 20 12:28:38 2023 -0400 Extend ALTER OPERATOR to allow setting more optimization attributes. Allow the COMMUTATOR, NEGATOR, MERGES, and HASHES attributes to be set by ALTER OPERATOR. However, we don't allow COMMUTATOR/NEGATOR to be changed once set, nor allow the MERGES/HASHES flags to be unset once set. Changes like that might invalidate plans already made, and dealing with the consequences seems like more trouble than it's worth. --> The main use-case we foresee for this is to allow addition of missed --> properties in extension update scripts, such as extending an existing --> operator to support hashing. So only transitions from not-set to set states seem very useful. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Thu, May 9, 2024 at 11:31:17AM +0100, Dagfinn Ilmari Mannsåker wrote: > Dagfinn Ilmari Mannsåker writes: > > > Bruce Momjian writes: > > > >> I have committed the first draft of the PG 17 release notes; you can > >> see the results here: > >> > >>https://momjian.us/pgsql_docs/release-17.html > > > > My name is listed twice in the "Improve psql tab completion" item. > > You can move one of them to "Track DEALLOCATE in pg_stat_statements", > which Michael and I co-authored. Yep, also my mistake, fixed. My apologies. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Thu, May 9, 2024 at 11:22:06AM +0100, Dagfinn Ilmari Mannsåker wrote: > Bruce Momjian writes: > > > I have committed the first draft of the PG 17 release notes; you can > > see the results here: > > > > https://momjian.us/pgsql_docs/release-17.html > > My name is listed twice in the "Improve psql tab completion" item. You did such a great job I wanted to list you twice. :-) Actually, the author list was so long I just didn't notice, fixed. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Thu, May 9, 2024 at 06:00:24PM +0800, jian he wrote: > On Thu, May 9, 2024 at 12:04 PM Bruce Momjian wrote: > > > > I have committed the first draft of the PG 17 release notes; you can > > see the results here: > > > > https://momjian.us/pgsql_docs/release-17.html > > > > another potential incompatibilities issue: > ALTER TABLE DROP PRIMARY KEY > > see: > https://www.postgresql.org/message-id/202404181849.6frtmajobe27%40alvherre.pgsql I see it now, and I see Alvaro Herrera saying: https://www.postgresql.org/message-id/202404181849.6frtmajobe27%40alvherre.pgsql > I wonder is there any incompatibility issue, or do we need to say something > about the new behavior when dropping a key column? --> Umm, yeah, maybe we should document it in ALTER TABLE DROP PRIMARY KEY --> and in the release notes to note the different behavior. However, I don't see it mentioned as a release note item in the commit message or mentioned in our docs. I suppose the release note text would be: Removing a PRIMARY KEY will remove the NOT NULL column specification Previously the NOT NULL specification would be retained. Do we have agreement that we want this release note item? -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Thu, May 9, 2024 at 12:18:44PM +0300, Aleksander Alekseev wrote: > Hi, > > > I have committed the first draft of the PG 17 release notes; you can > > see the results here: > > > > https://momjian.us/pgsql_docs/release-17.html > > > > It will be improved until the final release. The item count is 188, > > which is similar to recent releases: > > Thanks for working on this. > > I believe the part of the 64-bit XIDs patchset that was delivered in > PG17 is worth highlighting in "E.1.3.10. Source Code" section: > > 4ed8f0913bfd > 2cdf131c46e6 > 5a1dfde8334b > a60b8a58f435 > > All this can probably be summarized as one bullet "Index SLRUs by > 64-bit integers rather than by 32-bit ones" where the authors are: > Maxim Orlov, Aleksander Alekseev, Alexander Korotkov, Teodor Sigaev, > Nikita Glukhov, Pavel Borisov, Yura Sokolov. Wow, I try to only list source code items that have some user-facing impact, and I don't think these do. I do realize how important they are though. This gets into the balance of mentioning items _users_ need to know about, vs. important improvements that _we_ know about. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Thu, May 9, 2024 at 02:37:57PM +0800, Richard Guo wrote: > > On Thu, May 9, 2024 at 12:04 PM Bruce Momjian wrote: > > I have committed the first draft of the PG 17 release notes; you can > see the results here: > > https://momjian.us/pgsql_docs/release-17.html > > > Thanks for working on that. > > For this item: > > > Allow the optimizer to improve CTE plans by using the sort order of > columns referenced in earlier CTE clauses (Jian Guo) > > > I think you mean a65724dfa. The author should be 'Richard Guo'. Wow the CTE item above it was done by Jian Guo. I probably copied the text from the line above it, modified the description, but thought the author's name was the same, but it was not. Fixed. > And I'm wondering if it is more accurate to state it as "Allow the > optimizer to improve plans for the outer query by leveraging the sort > order of a CTE's output." > > I think maybe a similar revision can be applied to the item just above > this one. Okay, I went with this text: Allow the optimizer to improve CTE plans by considering the statistics of columns referenced in earlier row output clauses (Jian Guo, Tom Lane) Allow the optimizer to improve CTE plans by considering the sort order of columns referenced in earlier row output clauses (Richard Guo) I did not use "leveraging" because I am concerned non-native English speakers might find the term confusing. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Thu, May 9, 2024 at 02:17:12PM +0900, Masahiko Sawada wrote: > Hi, > > On Thu, May 9, 2024 at 1:03 PM Bruce Momjian wrote: > > > > I have committed the first draft of the PG 17 release notes; you can > > see the results here: > > > > https://momjian.us/pgsql_docs/release-17.html > > Thank you for working on that! > > I'd like to mention some of my works. I think we can add the vacuum > performance improvements by the following commits: > > - Add template for adaptive radix tree (ee1b30f1) > - Add TIDStore, to store sets of TIDs (ItemPointerData) efficiently > (30e144287) > - Use TidStore for dead tuple TIDs storage during lazy vacuum (667e65aac) Okay, I reworded the item, added authors, and added the commits: Allow vacuum to more efficiently remove and freeze tuples (John Naylor, Masahiko Sawada, Melanie Plageman) > Also, please consider the following item: > > - Improve eviction algorithm in ReorderBuffer using max-heap for many > subtransactions (5bec1d6bc) I looked at that item and I don't have a generic "make logical replication apply faster" item to merge it into, and many subtransactions seemed like enough of an edge-case that I didn't think mentioning it make sense. Can you see a good place to add it? > Finally, should we mention the following commit in the release note? > It's not a user-visible change but added a new regression test module. > > - Add tests for XID wraparound (e255b646a) I don't normally add testing infrastructure changes unless they are major. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Thu, May 9, 2024 at 04:53:38AM +, Bertrand Drouvot wrote: > Hi, > > On Thu, May 09, 2024 at 12:03:50AM -0400, Bruce Momjian wrote: > > I have committed the first draft of the PG 17 release notes; you can > > see the results here: > > > > https://momjian.us/pgsql_docs/release-17.html > > Thanks for working on that! > > > I welcome feedback. > > > Add system view pg_wait_events that reports wait event types (Michael > > Paquier) > > Michael is the committer for 1e68e43d3f, the author is me. Wow, thank you for finding that. The commit message is very clear so I don't know how I made that mistake. Fixed. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Thu, May 9, 2024 at 04:52:14PM +1200, David Rowley wrote: > On Thu, 9 May 2024 at 16:47, Muhammad Ikram wrote: > > A minor formatting issue in the start below. Bullet is not required here. > > This is a placeholder for the highlight features of v17 will go. > Bruce tends not to decide what those are all by himself. Yes, I already have so much of my opinion in the release notes that I prefer others to make that list, and to make the Acknowledgments list at the bottom. -- Bruce Momjian https://momjian.us EDB https://enterprisedb.com Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Thu, May 9, 2024 at 09:47:34AM +0500, Muhammad Ikram wrote:
> Hi Bruce,
>
> A minor formatting issue in the start below. Bullet is not required here.
>
>
> E.1.1. Overview
>
> PostgreSQL 17 contains many new features and enhancements, including:
>
> •
>
> The above items and other new features of PostgreSQL 17 are explained in more
> detail in the sections below.
That is just a place-holder. I changed the bullet text to be:
TO BE COMPLETED LATER
--
Bruce Momjian https://momjian.us
EDB https://enterprisedb.com
Only you can decide what is important to you.
Re: First draft of PG 17 release notes
On Thu, May 9, 2024 at 04:44:47PM +1200, David Rowley wrote:
> On Thu, 9 May 2024 at 16:04, Bruce Momjian wrote:
> > I welcome feedback. For some reason it was an easier job than usual.
>
> Thanks for working on that.
>
> > +2023-11-02 [cac169d68] Increase DEFAULT_FDW_TUPLE_COST from 0.01 to 0.2
>
> > +Double the default foreign data wrapper tuple cost (David Rowley, Umair
> > Shahid)
>
> That's 20x rather than 2x.
Oops, changed to:
Increase the default foreign data wrapper tuple cost (David
Rowley, Umair Shahid)
--
Bruce Momjian https://momjian.us
EDB https://enterprisedb.com
Only you can decide what is important to you.
