Re: Proposal: Filter irrelevant change before reassemble transactions during logical decoding

2024-05-21 Thread Ajin Cherian
On Tue, Mar 19, 2024 at 1:49 PM Ajin Cherian wrote: > > >> Of course you can, but this will only convert disk space into memory >> space. >> For details, please see the case in Email [1]. >> >> [1] >> https://www.postgresql.org/message-id/CAGfChW51P944nM5h0HTV9HistvVfwBxNaMt_s-OZ9t%3DuXz%2BZbg%4

Re: Path to unreverting "Allow planner to use Merge Append to efficiently implement UNION"

2024-05-21 Thread David Rowley
(Thanks for your review. I'm sorry I didn't have time and energy to respond properly until now) On Tue, 21 May 2024 at 23:48, Heikki Linnakangas wrote: > BTW, could the same machinery be used for INTERSECT as well? There was a > brief mention of that in the original thread, but I didn't understan

Re: Shared detoast Datum proposal

2024-05-21 Thread Andy Fan
Andy Fan writes: > Hi Robert, > >> Andy Fan asked me off-list for some feedback about this proposal. I >> have hesitated to comment on it for lack of having studied the matter >> in any detail, but since I've been asked for my input, here goes: > > Thanks for doing this! Since we have two tota

Re: First draft of PG 17 release notes

2024-05-21 Thread Masahiko Sawada
Hi, On Thu, May 9, 2024 at 1:03 PM Bruce Momjian wrote: > > I have committed the first draft of the PG 17 release notes; you can > see the results here: > > https://momjian.us/pgsql_docs/release-17.html > > It will be improved until the final release. The item count is 188, > which is s

Re: Shared detoast Datum proposal

2024-05-21 Thread Andy Fan
Hi Robert, > Andy Fan asked me off-list for some feedback about this proposal. I > have hesitated to comment on it for lack of having studied the matter > in any detail, but since I've been asked for my input, here goes: Thanks for doing this! Since we have two totally different designs betwee

Re: Skip adding row-marks for non target tables when result relation is foreign table.

2024-05-21 Thread Jeff Davis
On Mon, 2024-05-06 at 23:10 +0100, SAIKIRAN AVULA wrote: > I would like to bring to your attention an observation regarding the > planner's behavior for foreign table update/delete operations. It > appears that the planner adds rowmarks (ROW_MARK_COPY) for non-target > tables, which I believe is un

Re: problems with "Shared Memory and Semaphores" section of docs

2024-05-21 Thread Imseih (AWS), Sami
> Any concerns with doing something like this [0] for the back-branches? The > constant would be 6 instead of 7 on v14 through v16. As far as backpatching the present inconsistencies in the docs, [0] looks good to me. [0] https://postgr.es/m/attachment/160360/v1-0001-fix-kernel-resources-docs-on

Re: zlib detection in Meson on Windows broken?

2024-05-21 Thread Tristan Partin
On Tue May 21, 2024 at 10:04 AM CDT, Andres Freund wrote: Hi, On 2024-05-20 11:58:05 +0100, Dave Page wrote: > I have very little experience with Meson, and even less interpreting it's > logs, but it seems to me that it's not including the extra lib and include > directories when it runs the tes

Re: Path to unreverting "Allow planner to use Merge Append to efficiently implement UNION"

2024-05-21 Thread David Rowley
On Wed, 22 May 2024 at 05:36, Robert Haas wrote: > The consensus on pgsql-release was to unrevert this patch and commit > the fix now, rather than waiting for the next beta. However, the > consensus was also to push the un-revert as a separate commit from the > bug fix, rather than together as sug

Re: zlib detection in Meson on Windows broken?

2024-05-21 Thread Andrew Dunstan
On 2024-05-21 Tu 11:04, Andres Freund wrote: Hi, On 2024-05-20 11:58:05 +0100, Dave Page wrote: I have very little experience with Meson, and even less interpreting it's logs, but it seems to me that it's not including the extra lib and include directories when it runs the test compile, given

Re: SQL:2011 application time

2024-05-21 Thread Jeff Davis
On Tue, 2024-05-21 at 13:57 -0400, Robert Haas wrote: > What I think is less clear is what that means for temporal primary > keys. Right. My message was specifically a response to the concern that there was some kind of design flaw in the range types or exclusion constraints mechanisms. I don't

Re: libpq compression (part 3)

2024-05-21 Thread Jacob Champion
On Tue, May 21, 2024 at 12:08 PM Jacob Burroughs wrote: > I think I thought I was writing about something else when I wrote that > :sigh:. I think what I really should have written was a version of > the part below, which is that we use streaming decompression, only > decompress 8kb at a time, an

Re: libpq compression (part 3)

2024-05-21 Thread Jacob Burroughs
On Tue, May 21, 2024 at 1:24 PM Jacob Champion wrote: > > We absolutely have to document the risks and allow clients to be > written safely. But I think server-side controls on risky behavior > have proven to be generally more valuable, because the server > administrator is often in a better spot

Re: problems with "Shared Memory and Semaphores" section of docs

2024-05-21 Thread Nathan Bossart
On Fri, May 17, 2024 at 02:21:23PM -0500, Nathan Bossart wrote: > On Fri, May 17, 2024 at 12:48:37PM -0500, Nathan Bossart wrote: >> Cool. I'll at least fix the back-branches as-is, but I'll see about >> revamping this stuff for v18. > > Attached is probably the absolute least we should do for th

Re: libpq compression (part 3)

2024-05-21 Thread Jacob Burroughs
On Tue, May 21, 2024 at 1:39 PM Jacob Champion wrote: > > If the server doesn't reject compressed packets pre-authentication, > then case 2 isn't mitigated. (I haven't proven how risky that case is > yet, to be clear.) In other words: if the threat model is that a > client can attack us, we should

Re: First draft of PG 17 release notes

2024-05-21 Thread Robert Haas
On Tue, May 21, 2024 at 2:26 PM Melanie Plageman wrote: > For the vacuum WAL volume reduction, there were a bunch of smaller > projects throughout the last development year that I worked on that > were committed by different people and with different individual > benefits. Some changes caused vacu

Re: Convert node test compile-time settings into run-time parameters

2024-05-21 Thread Andres Freund
Hi, On 2024-05-20 09:28:39 +0200, Peter Eisentraut wrote: > - Performance? Looking for example at pg_parse_query() and its siblings, > they also check for other debugging settings like log_parser_stats in the > main code path, so it doesn't seem to be a concern. I don't think we can conclude tha

Re: libpq compression (part 3)

2024-05-21 Thread Jacob Champion
On Tue, May 21, 2024 at 9:14 AM Jacob Burroughs wrote: > On Tue, May 21, 2024 at 10:43 AM Jelte Fennema-Nio wrote: > > To help get everyone on the same page I wanted to list all the > > security concerns in one place: > > > > 1. Triggering excessive CPU usage before authentication, by asking for

Re: SQL:2011 application time

2024-05-21 Thread Isaac Morland
On Tue, 21 May 2024 at 13:57, Robert Haas wrote: What I think is less clear is what that means for temporal primary > keys. As Paul pointed out upthread, in every other case, a temporal > primary key is at least as unique as a regular primary key, but in > this case, it isn't. And someone might r

Re: First draft of PG 17 release notes

2024-05-21 Thread Melanie Plageman
On Tue, May 21, 2024 at 1:51 PM Robert Haas wrote: > > On Tue, May 21, 2024 at 12:27 PM Andres Freund wrote: > > To me that's the "General Performance" section. If somebody reading the > > release notes doesn't care about performance, they can just skip that > > section > > ([1]). I don't see w

Re: libpq compression (part 3)

2024-05-21 Thread Jacob Champion
On Tue, May 21, 2024 at 8:23 AM Jacob Burroughs wrote: > As currently implemented, the compression only applies to > CopyData/DataRow/Query messages, none of which should be involved in > authentication, unless I've really missed something in my > understanding. Right, but Robert has argued that

Re: SQL:2011 application time

2024-05-21 Thread Robert Haas
On Thu, May 16, 2024 at 7:22 PM Jeff Davis wrote: > An empty range does not "bypass" the an exclusion constraint. The > exclusion constraint has a documented meaning and it's enforced. > > Of course there are situations where an empty range doesn't make a lot > of sense. For many domains zero does

Re: First draft of PG 17 release notes

2024-05-21 Thread Robert Haas
On Tue, May 21, 2024 at 12:27 PM Andres Freund wrote: > To me that's the "General Performance" section. If somebody reading the > release notes doesn't care about performance, they can just skip that section > ([1]). I don't see why we wouldn't want to include the same level of detail > as for ot

Re: Path to unreverting "Allow planner to use Merge Append to efficiently implement UNION"

2024-05-21 Thread Robert Haas
On Tue, May 21, 2024 at 8:44 AM David Rowley wrote: > Thanks for having a look. I was planning to have the commit message > as per attached. I'd only split the patch for ease of review per > request of Tom. I should have mentioned that here. > > I would adjust the exact wording in the final parag

Re: First draft of PG 17 release notes

2024-05-21 Thread Peter Geoghegan
On Tue, May 21, 2024 at 12:27 PM Andres Freund wrote: > > I agree the impact of performance improvements are often greater than > > the average release note item. However, if people expect Postgres to be > > faster, is it important for them to know _why_ it is faster? > > Yes, it very often is.

Re: zlib detection in Meson on Windows broken?

2024-05-21 Thread Andres Freund
Hi, On 2024-05-20 11:58:05 +0100, Dave Page wrote: > I then attempt to build PostgreSQL: > > meson setup build > -Dextra_include_dirs=C:/build64/openssl/include,C:/build64/zlib/include > -Dextra_lib_dirs=C:/build64/openssl/lib,C:/build64/zlib/lib -Dssl=openssl > -Dzlib=enabled --prefix=c:/build6

Re: First draft of PG 17 release notes

2024-05-21 Thread Andres Freund
Hi, On 2024-05-21 09:27:20 -0700, Andres Freund wrote: > Also, the release notes are also not just important to users. I often go back > and look in the release notes to see when some some important change was made, > because sometimes it's harder to find it in the git log, due to sheer > volume.

Re: First draft of PG 17 release notes

2024-05-21 Thread Andres Freund
Hi, On 2024-05-18 11:13:54 -0400, Bruce Momjian wrote: > Please see the email I just posted. There are three goals we have to > adjust for: > > 1. short release notes so they are readable > 2. giving people credit for performance improvements > 3. showing people Postgres cares about performan

Re: First draft of PG 17 release notes

2024-05-21 Thread Andres Freund
Hi, On 2024-05-18 10:59:47 -0400, Bruce Momjian wrote: > On Wed, May 15, 2024 at 08:48:02PM -0700, Andres Freund wrote: > > +many. > > > > We're having this debate every release. I think the ongoing reticence to > > note > > performance improvements in the release notes is hurting Postgres. > > >

Re: libpq compression (part 3)

2024-05-21 Thread Jacob Burroughs
On Tue, May 21, 2024 at 10:43 AM Jelte Fennema-Nio wrote: > > To help get everyone on the same page I wanted to list all the > security concerns in one place: > > 1. Triggering excessive CPU usage before authentication, by asking for > very high compression levels > 2. Triggering excessive memory/

Re: libpq compression (part 3)

2024-05-21 Thread Jelte Fennema-Nio
On Mon, 20 May 2024 at 21:42, Jacob Champion wrote: > As Andrey points out, there was prior work done that started to take > this into account. I haven't reviewed it to see how good it is -- and > I think there are probably many use cases in which queries and tables > contain both private and atta

Re: broken tables on hot standby after migration on PostgreSQL 16 (3x times last month)

2024-05-21 Thread Andres Freund
On 2024-05-17 16:03:09 -0400, Peter Geoghegan wrote: > On Fri, May 17, 2024 at 3:50 PM Andres Freund wrote: > > You're saying that the data is correctly accessible on primaries, but broken > > on standbys? Is there any difference in how the page looks like on the > > primary > > vs standby? > >

Re: zlib detection in Meson on Windows broken?

2024-05-21 Thread Dave Page
On Tue, 21 May 2024 at 16:04, Andres Freund wrote: > Hi, > > On 2024-05-20 11:58:05 +0100, Dave Page wrote: > > I have very little experience with Meson, and even less interpreting it's > > logs, but it seems to me that it's not including the extra lib and > include > > directories when it runs t

Re: libpq compression (part 3)

2024-05-21 Thread Jacob Burroughs
On Mon, May 20, 2024 at 2:42 PM Jacob Champion wrote: > > I mean... you said it, not me. I'm trying not to rain on the parade > too much, because compression is clearly very valuable. But it makes > me really uncomfortable that we're reintroducing the compression > oracle (especially over the auth

Re: Possible Bug in relation_open

2024-05-21 Thread Tom Lane
Robert Haas writes: > On Tue, May 21, 2024 at 9:58 AM Pradeep Kumar > wrote: >> If the user tries to open the relation in RangeVar and NoLock mode calling >> table_openrv(relation, NoLock), it will internally call >> relation_openrv()-->relation_open(). In relation_open() we checking the >> A

Re: zlib detection in Meson on Windows broken?

2024-05-21 Thread Andres Freund
Hi, On 2024-05-20 11:58:05 +0100, Dave Page wrote: > I have very little experience with Meson, and even less interpreting it's > logs, but it seems to me that it's not including the extra lib and include > directories when it runs the test compile, given the command line it's > reporting: > > cl

Re: Possible Bug in relation_open

2024-05-21 Thread Robert Haas
On Tue, May 21, 2024 at 9:58 AM Pradeep Kumar wrote: > If the user tries to open the relation in RangeVar and NoLock mode calling > table_openrv(relation, NoLock), it will internally call > relation_openrv()-->relation_open(). In relation_open() we checking the > Assert(lockmode >= NoLock && lo

Re: zlib detection in Meson on Windows broken?

2024-05-21 Thread Dave Page
Hi On Tue, 21 May 2024 at 15:12, Sandeep Thakkar < sandeep.thak...@enterprisedb.com> wrote: > Hi Dave, > > > On Tue, May 21, 2024 at 3:12 PM Dave Page wrote: > >> Hi Sandeep, Nazir, >> >> On Tue, 21 May 2024 at 10:14, Nazir Bilal Yavuz >> wrote: >> >>> Hi, >>> >>> On Tue, 21 May 2024 at 10:20,

Re: zlib detection in Meson on Windows broken?

2024-05-21 Thread Sandeep Thakkar
Hi Dave, On Tue, May 21, 2024 at 3:12 PM Dave Page wrote: > Hi Sandeep, Nazir, > > On Tue, 21 May 2024 at 10:14, Nazir Bilal Yavuz > wrote: > >> Hi, >> >> On Tue, 21 May 2024 at 10:20, Sandeep Thakkar >> wrote: >> > >> > Hi Dave, >> > >> > Is the .pc file generated after the successful build

Possible Bug in relation_open

2024-05-21 Thread Pradeep Kumar
Hello Hackers, If the user tries to open the relation in RangeVar and NoLock mode calling *table_openrv(relation, NoLock), *it will internally call relation_openrv()-->relation_open(). In *relation_open() *we checking the Assert(lockmode >= NoLock && lockmode < MAX_LOCKMODES); , here we expecting

Re: ALTER TABLE SET ACCESS METHOD on partitioned tables

2024-05-21 Thread Justin Pryzby
It occurred to me that psql \dP+ should show the AM of partitioned tables (and other partitioned rels). Arguably, this could've been done when \dP was introduced in v12, but at that point would've shown the AM only for partitioned indexes. But it makes a lot of sense to do it now that partitioned t

Re: Reading timestamp values from Datums gives garbage values

2024-05-21 Thread Sushrut Shivaswamy
Thank you everyone for your responses. I was a bit thrown off by the timestamp value the first time I printed it by how small it was. The revelation that postgres TimestampTz uses an epoch (time zero) of 2000-01-01 helped clarify that value would indeed be smaller than regular UNIX epoch. In my c

Re: Avoid orphaned objects dependencies, take 3

2024-05-21 Thread Robert Haas
On Wed, May 15, 2024 at 6:31 AM Bertrand Drouvot wrote: > Please find attached v6 (only diff with v5 is moving the tests as suggested > above). I don't immediately know what to think about this patch. I've known about this issue for a long time, but I didn't think this was how we would fix it. I

Re: Path to unreverting "Allow planner to use Merge Append to efficiently implement UNION"

2024-05-21 Thread David Rowley
On Wed, 22 May 2024 at 00:35, Alvaro Herrera wrote: > > On 2024-May-21, David Rowley wrote: > > > I've attached 2 patches. > > > > 0001 is a simple revert of Tom's revert (7204f3591). > > 0002 fixes the issue reported by Hubert. > > I would like to request that you don't keep 0001's message as you

Re: Path to unreverting "Allow planner to use Merge Append to efficiently implement UNION"

2024-05-21 Thread Alvaro Herrera
On 2024-May-21, David Rowley wrote: > I've attached 2 patches. > > 0001 is a simple revert of Tom's revert (7204f3591). > 0002 fixes the issue reported by Hubert. I would like to request that you don't keep 0001's message as you have it here. It'd be more readable to take 66c0185a3d14's whole c

Re: libpq compression (part 3)

2024-05-21 Thread Robert Haas
On Mon, May 20, 2024 at 4:12 PM Magnus Hagander wrote: > That used to be the case in HTTP/1. But header compression was one of the > headline features of HTTP/2, which isn't exactly new anymore. But there's a > special algorithm, HPACK, for it. And then http/3 uses QPACK. Cloudflare has > a pre

Re: Convert node test compile-time settings into run-time parameters

2024-05-21 Thread Ranier Vilela
Em ter., 21 de mai. de 2024 às 09:25, Peter Eisentraut escreveu: > On 20.05.24 15:59, Tom Lane wrote: > > Peter Eisentraut writes: > >> This patch converts the compile-time settings > >> COPY_PARSE_PLAN_TREES > >> WRITE_READ_PARSE_PLAN_TREES > >> RAW_EXPRESSION_COVERAGE_TEST >

Re: Convert node test compile-time settings into run-time parameters

2024-05-21 Thread Peter Eisentraut
On 20.05.24 15:59, Tom Lane wrote: Peter Eisentraut writes: This patch converts the compile-time settings COPY_PARSE_PLAN_TREES WRITE_READ_PARSE_PLAN_TREES RAW_EXPRESSION_COVERAGE_TEST into run-time parameters debug_copy_parse_plan_trees debug_write_read_pars

Re: CREATE DATABASE with filesystem cloning

2024-05-21 Thread Ranier Vilela
Em ter., 21 de mai. de 2024 às 05:18, Nazir Bilal Yavuz escreveu: > Hi, > > On Thu, 16 May 2024 at 17:47, Robert Haas wrote: > > > > On Thu, May 16, 2024 at 9:43 AM Nazir Bilal Yavuz > wrote: > > > Actually, the documentation for the file_copy_method was mentioning > > > the things it controls;

Re: Path to unreverting "Allow planner to use Merge Append to efficiently implement UNION"

2024-05-21 Thread Heikki Linnakangas
On 21/05/2024 05:58, David Rowley wrote: Let this thread be for at least the coding portion of this or be my thread for this patch for the v18 cycle if the RMT rules in favour of keeping that code reverted for v17. I've attached 2 patches. 0001 is a simple revert of Tom's revert (7204f3591). 00

Re: Injection points: preloading and runtime arguments

2024-05-21 Thread Andrey M. Borodin
> On 21 May 2024, at 06:31, Michael Paquier wrote: > > So I agree that 0002 ought to call injection_init_shmem() when calling > injection_points_preload(), but it also seems to me that the test is > missing the fact that it should heat the backend cache to avoid the > allocations in the critic

Feature request: limiting isolation level choices

2024-05-21 Thread Tihrd Reed
Hello, Apologies if this is the wrong place to make such a request. It would be useful to have the ability to prevent clients from setting whatever isolation levels they choose. Specifically, it would be desirable to enforce SERIALIZABLE for all transactions since the serializable guarantee onl

Re: I have an exporting need...

2024-05-21 Thread Juan Hernández
Hi team! I read all your comments and this leads me to learn more. For me and my case would be useful, even there are other ways to solve this, but I may be wrong and just have to learn more about maintenance, backup and recovery tasks. What if when --separatetables clause is used, table definit

Re: PostgreSQL 17 Beta 1 release announcement draft

2024-05-21 Thread John Naylor
On Mon, May 20, 2024 at 8:41 PM Masahiko Sawada wrote: > > On Mon, May 20, 2024 at 8:47 PM Jonathan S. Katz wrote: > > > > On 5/20/24 2:58 AM, John Naylor wrote: > > > Hi Jon, > > > > > > Regarding vacuum "has shown up to a 6x improvement in overall time to > > > complete its work" -- I believe I

Re: Reading timestamp values from Datums gives garbage values

2024-05-21 Thread Aleksander Alekseev
Hi, > When trying to read the query response from the Datum, I get garbage values. > I've tried various types and none of them read the correct value. > ``` > > Datum current_timestamp = SPI_getbinval(SPI_tuptable->vals[i], > SPI_tuptable->tupdesc, 5, &isnull); > > double current_time = DatumGetF

Re: zlib detection in Meson on Windows broken?

2024-05-21 Thread Dave Page
Hi Sandeep, Nazir, On Tue, 21 May 2024 at 10:14, Nazir Bilal Yavuz wrote: > Hi, > > On Tue, 21 May 2024 at 10:20, Sandeep Thakkar > wrote: > > > > Hi Dave, > > > > Is the .pc file generated after the successful build of zlib? If yes, > then meson should be able to detect the installation ideall

How to declare GIN index on array type column when bootstrap?

2024-05-21 Thread Zhang Mingli
 Hi, all I want to add some columns of int(or Oid) array and declare GIN index for it in catalog when bootstrap. But current catalogs all use btree, I tried to declare a GIN index but failed, ex: pg_class.h ``` CATALOG(pg_class ... Int32 my_new_column[1] BKI_DEFAULT(_null_); ... } FormData_pg

Re: zlib detection in Meson on Windows broken?

2024-05-21 Thread Nazir Bilal Yavuz
Hi, On Tue, 21 May 2024 at 10:20, Sandeep Thakkar wrote: > > Hi Dave, > > Is the .pc file generated after the successful build of zlib? If yes, then > meson should be able to detect the installation ideally If meson is not able to find the .pc file automatically, using 'meson setup ... --pkg-co

Re: POC: GROUP BY optimization

2024-05-21 Thread Andrei Lepikhov
On 20/5/2024 15:54, jian he wrote: As mentioned previously, both A and B can use Incremental Sort, set_cheapest will choose the A instead of B, because A step generated path **first** satisfies increment sort. Yeah, I agree with your analysis. Looking into the cost_incremental_sort, I see that w

Re: CREATE DATABASE with filesystem cloning

2024-05-21 Thread Nazir Bilal Yavuz
Hi, On Thu, 16 May 2024 at 17:47, Robert Haas wrote: > > On Thu, May 16, 2024 at 9:43 AM Nazir Bilal Yavuz wrote: > > Actually, the documentation for the file_copy_method was mentioning > > the things it controls; I converted it to an itemized list now. Also, > > changed the comment to: 'Further

Re: Avoid orphaned objects dependencies, take 3

2024-05-21 Thread Bertrand Drouvot
Hi, On Sun, May 19, 2024 at 11:00:00AM +0300, Alexander Lakhin wrote: > Hello Bertrand, > > Probably, it's worth to avoid ERRCODE_INTERNAL_ERROR here in light of [1] > and [2], as these errors are not that abnormal (not Assert-like). > > [1] https://www.postgresql.org/message-id/Zic_GNgos5sMxKoa

Re: zlib detection in Meson on Windows broken?

2024-05-21 Thread Sandeep Thakkar
Hi Dave, Is the .pc file generated after the successful build of zlib? If yes, then meson should be able to detect the installation ideally On Mon, May 20, 2024 at 4:28 PM Dave Page wrote: > Hi > > I'm working on updating the build of PostgreSQL that pgAdmin uses in its > Windows installers to