Re: A qsort template

2022-04-18 Thread David Rowley
Thanks for looking at this. On Tue, 19 Apr 2022 at 02:11, John Naylor wrote: > IIUC, this function is called by tuplesort_begin_common, which in turn > is called by tuplesort_begin_{heap, indexes, etc}. The latter callers > set the onlyKey and now oneKeySort variables as appropriate, and >

Re: Replace open mode with PG_BINARY_R/W/A macros

2022-04-18 Thread Japin Li
On Mon, 18 Apr 2022 at 22:41, Tom Lane wrote: > Japin Li writes: >> I found we defined PG_BINARY_R/W/A macros for opening files, however, >> there are some places use the constant strings. IMO we should use >> those macros instead of constant strings. Here is a patch for it. >> Any thoughts?

Re: Non-replayable WAL records through overflows and >MaxAllocSize lengths

2022-04-18 Thread Michael Paquier
On Mon, Apr 18, 2022 at 05:48:50PM +0200, Matthias van de Meent wrote: > Seeing that the busiest time for PG15 - the last commitfest before the > feature freeze - has passed, could someone take another look at this? The next minor release is three weeks away, so now would be a good time to get

Re: pg_receivewal fail to streams when the partial file to write is not fully initialized present in the wal receiver directory

2022-04-18 Thread Michael Paquier
On Mon, Apr 18, 2022 at 02:50:17PM +0530, Bharath Rupireddy wrote: > Thanks for sharing this. Will the write operations (in > dir_open_for_write) for PG_COMPRESSION_GZIP and PG_COMPRESSION_LZ4 > take longer compared to prepadding for non-compressed files? The first write operations for gzip and

Re: TRAP: FailedAssertion("tabstat->trans == trans", File: "pgstat_relation.c", Line: 508

2022-04-18 Thread Erik Rijkers
Op 19-04-2022 om 02:15 schreef Kyotaro Horiguchi: At Mon, 18 Apr 2022 10:57:02 +0200, Erikjan Rijkers wrote in Hm. Just now I've recompiled and retried and after 5 runs got the same crash. Then tried on another machine (also old, I'm afraid), and built 1a8b11053 and ran the same thing. That

Re: Column Filtering in Logical Replication

2022-04-18 Thread Amit Kapila
On Tue, Apr 19, 2022 at 6:58 AM Masahiko Sawada wrote: > > On Mon, Apr 18, 2022 at 8:04 PM Amit Kapila wrote: > > > > On Thu, Apr 14, 2022 at 9:09 AM Amit Kapila wrote: > > > > > > On Thu, Apr 14, 2022 at 8:32 AM Masahiko Sawada > > > wrote: > > > > > > > > > > The other part of the puzzle is

Re: using an end-of-recovery record in all cases

2022-04-18 Thread Amul Sul
On Tue, Apr 19, 2022 at 2:14 AM Robert Haas wrote: > > On Sat, Jan 15, 2022 at 11:52 PM Julien Rouhaud wrote: > > The cfbot reports that this version of the patch doesn't apply anymore: > > Here is a new version of the patch which, unlike v1, I think is > something we could seriously consider

Re: Handle infinite recursion in logical replication setup

2022-04-18 Thread Peter Smith
I checked the latest v9-0001 patch. Below are my review comments. Other than these few trivial comments this 0001 patch looks good to me. ~~~ 1. src/backend/replication/pgoutput/pgoutput.c - whitespace @@ -1696,6 +1714,10 @@ static bool pgoutput_origin_filter(LogicalDecodingContext *ctx,

Re: Fix NULL pointer reference in _outPathTarget()

2022-04-18 Thread Richard Guo
On Tue, Apr 19, 2022 at 2:53 AM Tom Lane wrote: > > A semantics-preserving conversion would have looked something like > > if (node->sortgrouprefs) > WRITE_INDEX_ARRAY(sortgrouprefs, list_length(node->exprs)); > > I suppose that Peter was trying to remove special cases from the >

Re: Postgres perl module namespace

2022-04-18 Thread Michael Paquier
On Mon, Apr 18, 2022 at 01:59:23PM -0400, Andrew Dunstan wrote: > On 2022-04-18 Mo 13:43, Tom Lane wrote: >> I doubt that just plopping the new Cluster.pm in alongside the old >> file could work --- wouldn't the two modules need to share state >> somehow? > > No, I think we could probably just

Re: pgsql: Add TAP test for archive_cleanup_command and recovery_end_comman

2022-04-18 Thread Michael Paquier
On Tue, Apr 19, 2022 at 09:45:11AM +1200, Thomas Munro wrote: > Delayed response to the question on how I did that, because it was a 4 > day weekend down here and I got distracted by sunshine... Happy Easter. > I think that sort of thing actually worked when I tried it on a > beefier

RE: Logical replication timeout problem

2022-04-18 Thread wangw.f...@fujitsu.com
On Mon, Apr 19, 2022 at 9:32 AM Masahiko Sawada wrote: > Thank you for updating the patch. Thanks for your comments. > + * For a large transaction, if we don't send any change to the > + downstream for a > + * long time(exceeds the wal_receiver_timeout of standby) then it can > timeout. > + *

Re: Add --{no-,}bypassrls flags to createuser

2022-04-18 Thread Kyotaro Horiguchi
Thanks! At Mon, 18 Apr 2022 09:59:48 -0400, Robert Haas wrote in > On Fri, Apr 15, 2022 at 2:33 AM Kyotaro Horiguchi > wrote: > > > printf(_(" -b, --belongs-to=ROLE new role will be a member of this > > > role\n")); > > > > + printf(_(" -m, --membership=ROLE this role will be

Re: BufferAlloc: don't take two simultaneous locks

2022-04-18 Thread Kyotaro Horiguchi
At Mon, 18 Apr 2022 09:53:42 -0400, Robert Haas wrote in > On Fri, Apr 15, 2022 at 4:29 AM Kyotaro Horiguchi > wrote: > > The patch removes buftable entry frist then either inserted again or > > returned to freelist. I don't understand how it can be in both > > buftable and freelist.. What

Re: Logical replication timeout problem

2022-04-18 Thread Masahiko Sawada
On Mon, Apr 18, 2022 at 3:16 PM wangw.f...@fujitsu.com wrote: > > On Mon, Apr 18, 2022 at 00:35 PM Masahiko Sawada > wrote: > > On Mon, Apr 18, 2022 at 1:01 PM Amit Kapila wrote: > > > > > > On Thu, Apr 14, 2022 at 5:50 PM Masahiko Sawada > > wrote: > > > > > > > > On Wed, Apr 13, 2022 at

Re: Column Filtering in Logical Replication

2022-04-18 Thread Masahiko Sawada
On Mon, Apr 18, 2022 at 8:04 PM Amit Kapila wrote: > > On Thu, Apr 14, 2022 at 9:09 AM Amit Kapila wrote: > > > > On Thu, Apr 14, 2022 at 8:32 AM Masahiko Sawada > > wrote: > > > > > > > > The other part of the puzzle is the below check in the code: > > > > /* > > > > * If we reached the sync

Re: Why does pg_class.reltuples count only live tuples in indexes (after VACUUM runs)?

2022-04-18 Thread Peter Geoghegan
On Mon, Apr 18, 2022 at 1:12 PM Peter Geoghegan wrote: > I would argue that it would be correct for the first time -- at least > if we take the behavior within heapam_index_build_range_scan (and > everywhere else) as authoritative. That's a feature, not a bug. Attached draft patch shows what I

Re: make MaxBackends available in _PG_init

2022-04-18 Thread Tom Lane
Nathan Bossart writes: > I'm looking for a clean way to ERROR if someone attempts to call > RequestAddinShmemSpace() or RequestNamedLWLockTranche() outside of the > hook. Currently, we are using static variables in ipci.c and lwlock.c to > silently ignore invalid requests. I could add a new

Re: TRAP: FailedAssertion("tabstat->trans == trans", File: "pgstat_relation.c", Line: 508

2022-04-18 Thread Kyotaro Horiguchi
At Mon, 18 Apr 2022 10:57:02 +0200, Erikjan Rijkers wrote in > Hm. Just now I've recompiled and retried and after 5 runs got the > same crash. Then tried on another machine (also old, I'm afraid), > and built 1a8b11053 and ran the same thing. That failed on the first > try, and made core dump

Re: make MaxBackends available in _PG_init

2022-04-18 Thread Nathan Bossart
On Mon, Apr 18, 2022 at 07:33:54PM -0400, Tom Lane wrote: > Nathan Bossart writes: >> I noticed that requests for more LWLocks follow a similar pattern as >> regular shared memory requests, and I figured that we would want to do >> something similar for those, but I wasn't sure exactly how to

Re: make MaxBackends available in _PG_init

2022-04-18 Thread Tom Lane
Nathan Bossart writes: > I noticed that requests for more LWLocks follow a similar pattern as > regular shared memory requests, and I figured that we would want to do > something similar for those, but I wasn't sure exactly how to proceed. I > saw two options: 1) use shmem_request_hook for both

Re: PG DOCS - logical replication filtering

2022-04-18 Thread Peter Smith
On Mon, Apr 18, 2022 at 3:53 PM Amit Kapila wrote: > > On Thu, Apr 14, 2022 at 8:55 AM Amit Kapila wrote: > > > > On Thu, Apr 14, 2022 at 1:29 AM Euler Taveira wrote: > > > > > > On Wed, Apr 13, 2022, at 12:24 AM, Peter Smith wrote: > > > > > > PSA patch v10 which addresses the remaining review

Re: make MaxBackends available in _PG_init

2022-04-18 Thread Nathan Bossart
On Thu, Apr 14, 2022 at 12:39:46PM -0400, Tom Lane wrote: > Robert Haas writes: >> On Thu, Apr 14, 2022 at 12:22 PM Nathan Bossart >> wrote: I'd be in favor of a hard break. > >>> Yeah, this is a good point. If we're okay with breaking existing >>> extensions like this, I will work on a

Re: [Proposal] vacuumdb --schema only

2022-04-18 Thread Nathan Bossart
On Thu, Apr 14, 2022 at 10:27:46PM +0200, Gilles Darold wrote: > Attached v8 of the patch that tries to address the remarks above, fixes > patch apply failure to master and replace calls to pg_log_error+exit > with pg_fatal. Thanks for the new patch. > +enum trivalue schema_is_exclude =

Re: pgsql: Add TAP test for archive_cleanup_command and recovery_end_comman

2022-04-18 Thread Thomas Munro
On Mon, Apr 18, 2022 at 11:49 AM Michael Paquier wrote: > On Sun, Apr 17, 2022 at 10:56:08AM -0400, Andrew Dunstan wrote: > > I don't really think it's Cluster.pm's business to deal with that. It > > takes an install path as given either explicitly or implicitly. > > > > It shouldn't be too hard

Hash index build performance tweak from sorting

2022-04-18 Thread Simon Riggs
Hash index pages are stored in sorted order, but we don't prepare the data correctly. We sort the data as the first step of a hash index build, but we forget to sort the data by hash as well as by hash bucket. This causes the normal insert path to do extra pushups to put the data in the correct

Re: Dump/Restore of non-default PKs

2022-04-18 Thread Simon Riggs
On Mon, 18 Apr 2022 at 21:48, Tom Lane wrote: > > "David G. Johnston" writes: > > On Mon, Apr 18, 2022 at 1:00 PM Simon Riggs > > wrote: > >> I propose that we change pg_dump so that when it creates a PK it does > >> so in 2 commands: > >> 1. CREATE [UNIQUE] INDEX iname ... > >> 2. ALTER TABLE

Re: Dump/Restore of non-default PKs

2022-04-18 Thread David G. Johnston
On Mon, Apr 18, 2022 at 1:48 PM Tom Lane wrote: > "David G. Johnston" writes: > > On Mon, Apr 18, 2022 at 1:00 PM Simon Riggs < > simon.ri...@enterprisedb.com> > > wrote: > >> I propose that we change pg_dump so that when it creates a PK it does > >> so in 2 commands: > >> 1. CREATE [UNIQUE]

Re: avoid multiple hard links to same WAL file after a crash

2022-04-18 Thread Greg Stark
The readdir interface allows processes to be in the middle of reading a directory and unless a kernel was happy to either materialize the entire directory list when the readdir starts, or lock the entire directory against modification for the entire time the a process has a readdir fd open it's

Re: Dump/Restore of non-default PKs

2022-04-18 Thread Tom Lane
"David G. Johnston" writes: > On Mon, Apr 18, 2022 at 1:00 PM Simon Riggs > wrote: >> I propose that we change pg_dump so that when it creates a PK it does >> so in 2 commands: >> 1. CREATE [UNIQUE] INDEX iname ... >> 2. ALTER TABLE .. ADD PRIMARY KEY USING INDEX iname; > Why not just get rid

Re: using an end-of-recovery record in all cases

2022-04-18 Thread Robert Haas
On Sat, Jan 15, 2022 at 11:52 PM Julien Rouhaud wrote: > The cfbot reports that this version of the patch doesn't apply anymore: Here is a new version of the patch which, unlike v1, I think is something we could seriously consider applying (not before v16, of course). It now removes

Re: Dump/Restore of non-default PKs

2022-04-18 Thread Tom Lane
Simon Riggs writes: > I propose that we change pg_dump so that when it creates a PK it does > so in 2 commands: > 1. CREATE [UNIQUE] INDEX iname ... > 2. ALTER TABLE .. ADD PRIMARY KEY USING INDEX iname; > Step > (1) recreates the index, respecting its AM, even if that is not a btree > (2) works

Re: Postgres perl module namespace

2022-04-18 Thread Mark Dilger
> On Apr 18, 2022, at 1:19 PM, Andrew Dunstan wrote: > > that seems quite separate from the present issue. Thanks for the clarification. I agree, given your comments, that it is unrelated to this thread. — Mark Dilger EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL

Re: Dump/Restore of non-default PKs

2022-04-18 Thread David G. Johnston
On Mon, Apr 18, 2022 at 1:00 PM Simon Riggs wrote: > At the moment you cannot create a unique index other than a btree. (As > discussed on other threads, I am pursuing unique hash indexes for > PostgreSQL, one step at a time). > You get "ERROR index foo_idx is not a btree" > > According to

Re: Postgres perl module namespace

2022-04-18 Thread Andrew Dunstan
On 2022-04-18 Mo 15:46, Mark Dilger wrote: > >> On Apr 18, 2022, at 10:59 AM, Andrew Dunstan wrote: >> >> No, I think we could probably just port the whole of src/test/PostreSQL >> back if required, and have it live alongside the old modules. Each TAP >> test is a separate miracle - see

Re: Why does pg_class.reltuples count only live tuples in indexes (after VACUUM runs)?

2022-04-18 Thread Peter Geoghegan
On Mon, Apr 18, 2022 at 1:10 PM Tom Lane wrote: > The places where index AMs refer to num_heap_tuples seem to be using > it as a ceiling on estimated index tuple counts. Given that we should > be counting dead index entries, redefining it as you suggest would be > wrong. I would argue that it

Re: Why does pg_class.reltuples count only live tuples in indexes (after VACUUM runs)?

2022-04-18 Thread Tom Lane
Peter Geoghegan writes: > I think that this doesn't really belong here; new_rel_tuples should > only be used for VACUUM VERBOSE/server log output, once we return to > heap_vacuum_rel from lazy_scan_heap. We should use > vacrel->new_live_tuples as our IndexVacuumInfo.num_heap_tuples value > in the

Re: TRAP: FailedAssertion("HaveRegisteredOrActiveSnapshot()", File: "toast_internals.c", Line: 670, PID: 19403)

2022-04-18 Thread Tom Lane
Andres Freund writes: > On 2022-04-17 11:51:58 -0400, Tom Lane wrote: >> The fact that we have a snapshot at the instant of fetch doesn't prove >> that it existed continually since we fetched the toast reference, >> which seems to be the condition we actually need to assure. > Right. BTW, after

Re: Why does pg_class.reltuples count only live tuples in indexes (after VACUUM runs)?

2022-04-18 Thread Peter Geoghegan
On Mon, Apr 18, 2022 at 12:41 PM Tom Lane wrote: > If the planner looks at index reltuples at all, it's doing so > for cost estimation purposes, where the count including dead > entries is probably the right thing to use. Then why does heapam_index_build_range_scan do it the other way around? I

Re: Preventing indirection for IndexPageGetOpaque for known-size page special areas

2022-04-18 Thread Bruce Momjian
On Thu, Apr 7, 2022 at 03:11:24PM -0400, Robert Haas wrote: > On Thu, Apr 7, 2022 at 2:43 PM Peter Geoghegan wrote: > > But if we were in a green-field situation we'd probably not want to > > use up several bytes for a nonse anyway. You said so yourself. > > I don't know what statement of mine

Dump/Restore of non-default PKs

2022-04-18 Thread Simon Riggs
At the moment you cannot create a unique index other than a btree. (As discussed on other threads, I am pursuing unique hash indexes for PostgreSQL, one step at a time). You get "ERROR index foo_idx is not a btree" According to parse_utilcmd.c line 2310, this is because it would break pg_dump,

Re: Postgres perl module namespace

2022-04-18 Thread Mark Dilger
> On Apr 18, 2022, at 10:59 AM, Andrew Dunstan wrote: > > No, I think we could probably just port the whole of src/test/PostreSQL > back if required, and have it live alongside the old modules. Each TAP > test is a separate miracle - see comments elsewhere about port > assignment in parallel

Re: Why does pg_class.reltuples count only live tuples in indexes (after VACUUM runs)?

2022-04-18 Thread Tom Lane
Peter Geoghegan writes: > On Mon, Apr 18, 2022 at 12:15 PM Tom Lane wrote: >> Huh? This is not pg_class.reltuples. If an index AM wants that, it >> knows where to find it. > It's not, but it is how we calculate > IndexBulkDeleteResult.num_index_tuples, which is related. Well, the number of

Re: pg14 psql broke \d datname.nspname.relname

2022-04-18 Thread Mark Dilger
> On Apr 8, 2022, at 4:11 AM, Robert Haas wrote: > > I don't personally see how we're going to come out ahead with that > approach, but if you or Tom or someone else want to put something > together, that's fine with me. I'm not stuck on this approach, I just > don't see how we come out ahead

Re: Postgres perl module namespace

2022-04-18 Thread Andrew Dunstan
On 2022-04-18 Mo 14:07, Tom Lane wrote: > Andrew Dunstan writes: >> No, I think we could probably just port the whole of src/test/PostreSQL >> back if required, and have it live alongside the old modules. Each TAP >> test is a separate miracle - see comments elsewhere about port >> assignment

Re: Why does pg_class.reltuples count only live tuples in indexes (after VACUUM runs)?

2022-04-18 Thread Peter Geoghegan
On Mon, Apr 18, 2022 at 12:15 PM Tom Lane wrote: > > I don't see why it makes sense to treat indexes differently here. Why > > allow the special case? Why include dead tuples like this? > > The index has presumably got entries corresponding to dead tuples, > so that the number of entries it has

Re: Why does pg_class.reltuples count only live tuples in indexes (after VACUUM runs)?

2022-04-18 Thread Tom Lane
Peter Geoghegan writes: > Commit 7c91a0364f standardized the approach we take to estimating > pg_class.reltuples, so that everybody agrees on what that means. > Follow-up work by commit 3d351d91 defined a pg_class.reltuples of -1 > as "unknown, probably never vacuumed". > The former commit added

Re: avoid multiple hard links to same WAL file after a crash

2022-04-18 Thread Tom Lane
Michael Paquier writes: > On Fri, Apr 08, 2022 at 09:00:36PM -0400, Robert Haas wrote: >> I wonder if this is really true. I thought rename() was supposed to be >> atomic. > Not always. For example, some old versions of MacOS have a non-atomic > implementation of rename(), like prairiedog with

Why does pg_class.reltuples count only live tuples in indexes (after VACUUM runs)?

2022-04-18 Thread Peter Geoghegan
Commit 7c91a0364f standardized the approach we take to estimating pg_class.reltuples, so that everybody agrees on what that means. Follow-up work by commit 3d351d91 defined a pg_class.reltuples of -1 as "unknown, probably never vacuumed". The former commit added this code and comment to

Re: Fix NULL pointer reference in _outPathTarget()

2022-04-18 Thread Tom Lane
Richard Guo writes: > The array sortgrouprefs[] inside PathTarget might be NULL if we have not > identified sort/group columns in this tlist. In that case we would have > a NULL pointer reference in _outPathTarget() when trying to print > sortgrouprefs[] with WRITE_INDEX_ARRAY as we are using the

Re: avoid multiple hard links to same WAL file after a crash

2022-04-18 Thread Nathan Bossart
On Mon, Apr 18, 2022 at 04:48:35PM +0900, Michael Paquier wrote: > Saying that, it would be nice to see durable_rename_excl() gone as it > has created quite a bit of pain for us in the past years. Yeah, I think this is the right thing to do. Patch upthread [0]. For back-branches, I suspect

Re: Postgres perl module namespace

2022-04-18 Thread Tom Lane
Andrew Dunstan writes: > No, I think we could probably just port the whole of src/test/PostreSQL > back if required, and have it live alongside the old modules. Each TAP > test is a separate miracle - see comments elsewhere about port > assignment in parallel TAP tests. > But that would mean we

Re: Postgres perl module namespace

2022-04-18 Thread Andrew Dunstan
On 2022-04-18 Mo 13:43, Tom Lane wrote: > Andrew Dunstan writes: >> On 2022-04-18 Mo 11:52, Noah Misch wrote: >>> On Mon, Apr 18, 2022 at 07:15:30AM -0700, Andres Freund wrote: I just, again, tried to backport a test as part of a bugfix. The renaming between 14 and 15 makes that task

Re: Postgres perl module namespace

2022-04-18 Thread Tom Lane
Andrew Dunstan writes: > On 2022-04-18 Mo 11:52, Noah Misch wrote: >> On Mon, Apr 18, 2022 at 07:15:30AM -0700, Andres Freund wrote: >>> I just, again, tried to backport a test as part of a bugfix. The >>> renaming between 14 and 15 makes that task almost comically harder. The >>> only way I see

Re: pgsql: Add TAP test for archive_cleanup_command and recovery_end_comman

2022-04-18 Thread Andrew Dunstan
On 2022-04-17 Su 19:49, Michael Paquier wrote: > On Sun, Apr 17, 2022 at 10:56:08AM -0400, Andrew Dunstan wrote: >> I don't really think it's Cluster.pm's business to deal with that. It >> takes an install path as given either explicitly or implicitly. >> >> It shouldn't be too hard to get

Re: Postgres perl module namespace

2022-04-18 Thread Andrew Dunstan
On 2022-04-18 Mo 11:52, Noah Misch wrote: > On Mon, Apr 18, 2022 at 07:15:30AM -0700, Andres Freund wrote: >> I just, again, tried to backport a test as part of a bugfix. The >> renaming between 14 and 15 makes that task almost comically harder. The >> only way I see of dealing with that for the

Re: TRAP: FailedAssertion("HaveRegisteredOrActiveSnapshot()", File: "toast_internals.c", Line: 670, PID: 19403)

2022-04-18 Thread Tom Lane
Robert Haas writes: > I wasn't really taking a position either way about timing. If we can > demonstrate that things other than HaveRegisteredOrActiveSnapshot() > itself are misbehaving, then I think fixes for those bugs are > potentially back-patchable no matter where we are in the release >

Re: subscribe hackers

2022-04-18 Thread John Naylor
On Mon, Apr 18, 2022 at 6:54 PM 汪洋 wrote: > > subscribe pgsql-hackers Hi, this mailing list is not managed by subject line. To subscribe, please visit https://lists.postgresql.org/ -- John Naylor EDB: http://www.enterprisedb.com

Re: TRAP: FailedAssertion("HaveRegisteredOrActiveSnapshot()", File: "toast_internals.c", Line: 670, PID: 19403)

2022-04-18 Thread Robert Haas
On Mon, Apr 18, 2022 at 11:26 AM Tom Lane wrote: > Robert Haas writes: > > I agree that it's a little unclear. In general, I think if we're going > > to blow up and die, doing it closer to the place where the problem is > > happening is for the best. On the other hand, if in most practical > >

Re: Postgres perl module namespace

2022-04-18 Thread Noah Misch
On Mon, Apr 18, 2022 at 07:15:30AM -0700, Andres Freund wrote: > I just, again, tried to backport a test as part of a bugfix. The > renaming between 14 and 15 makes that task almost comically harder. The > only way I see of dealing with that for the next 5 years is to just > never backpatch tests

Re: Non-replayable WAL records through overflows and >MaxAllocSize lengths

2022-04-18 Thread Matthias van de Meent
Seeing that the busiest time for PG15 - the last commitfest before the feature freeze - has passed, could someone take another look at this? The changes that were requested by Heikki and Andres have been merged into patch v3, and I think it would be nice to fix this security issue in the upcoming

Re: TRAP: FailedAssertion("HaveRegisteredOrActiveSnapshot()", File: "toast_internals.c", Line: 670, PID: 19403)

2022-04-18 Thread Tom Lane
Robert Haas writes: > I agree that it's a little unclear. In general, I think if we're going > to blow up and die, doing it closer to the place where the problem is > happening is for the best. On the other hand, if in most practical > cases we're going to stumble through and get the right answer

Re: pg_walcleaner - new tool to detect, archive and delete the unneeded wal files (was Re: pg_archivecleanup - add the ability to detect, archive and delete the unneeded wal files on the primary)

2022-04-18 Thread Stephen Frost
Greeting, * Bharath Rupireddy (bharath.rupireddyforpostg...@gmail.com) wrote: > On Mon, Apr 18, 2022 at 7:41 PM Stephen Frost wrote: > > * Bharath Rupireddy (bharath.rupireddyforpostg...@gmail.com) wrote: > > > Thanks for the comments. Here's a new tool called pg_walcleaner which > > > basically

Re: TRAP: FailedAssertion("HaveRegisteredOrActiveSnapshot()", File: "toast_internals.c", Line: 670, PID: 19403)

2022-04-18 Thread Robert Haas
On Mon, Apr 18, 2022 at 10:53 AM Tom Lane wrote: > Robert Haas writes: > > I still think it would be better to have GetOldestSnapshot() be > > smarter and refuse to return the catalog snapshot. For one thing, that > > way we'd be testing for the problem case in non-assert builds also. > > I was

Re: pg_walcleaner - new tool to detect, archive and delete the unneeded wal files (was Re: pg_archivecleanup - add the ability to detect, archive and delete the unneeded wal files on the primary)

2022-04-18 Thread Bharath Rupireddy
On Mon, Apr 18, 2022 at 7:41 PM Stephen Frost wrote: > > Greetings, > > * Bharath Rupireddy (bharath.rupireddyforpostg...@gmail.com) wrote: > > Thanks for the comments. Here's a new tool called pg_walcleaner which > > basically deletes (optionally archiving before deletion) the unneeded > > WAL

Re: TRAP: FailedAssertion("HaveRegisteredOrActiveSnapshot()", File: "toast_internals.c", Line: 670, PID: 19403)

2022-04-18 Thread Tom Lane
Robert Haas writes: > I still think it would be better to have GetOldestSnapshot() be > smarter and refuse to return the catalog snapshot. For one thing, that > way we'd be testing for the problem case in non-assert builds also. I was wondering about that too. On the other hand, given that we

Re: TRAP: FailedAssertion("HaveRegisteredOrActiveSnapshot()", File: "toast_internals.c", Line: 670, PID: 19403)

2022-04-18 Thread Tom Lane
Andres Freund writes: > On 2022-04-17 11:51:58 -0400, Tom Lane wrote: >> The fact that we have a snapshot at the instant of fetch doesn't prove >> that it existed continually since we fetched the toast reference, >> which seems to be the condition we actually need to assure. > Right. >> (And

Re: Crash in new pgstats code

2022-04-18 Thread Andres Freund
Hi, On 2022-04-18 22:45:07 +1200, Thomas Munro wrote: > On Mon, Apr 18, 2022 at 7:19 PM Michael Paquier wrote: > > On Sat, Apr 16, 2022 at 02:36:33PM -0700, Andres Freund wrote: > > > which I haven't seen locally. Looks like we have some race between > > > startup process and walreceiver? That

Re: Postgres perl module namespace

2022-04-18 Thread Andres Freund
Hi, On 2022-04-18 10:26:15 -0400, Tom Lane wrote: > Andres Freund writes: > > I just, again, tried to backport a test as part of a bugfix. The > > renaming between 14 and 15 makes that task almost comically harder. The > > only way I see of dealing with that for the next 5 years is to just > >

Re: TRAP: FailedAssertion("HaveRegisteredOrActiveSnapshot()", File: "toast_internals.c", Line: 670, PID: 19403)

2022-04-18 Thread Robert Haas
On Mon, Apr 18, 2022 at 10:39 AM Andres Freund wrote: > Right. It's better than what was there before though - I added > HaveRegisteredOrActiveSnapshot() in the course of > 7c38ef2a5d6cf6d8dc3834399d7a1c364d64ce64. Where the problem was that we > didn't have *any* snapshot other than the catalog

Re: Replace open mode with PG_BINARY_R/W/A macros

2022-04-18 Thread Tom Lane
Japin Li writes: > I found we defined PG_BINARY_R/W/A macros for opening files, however, > there are some places use the constant strings. IMO we should use > those macros instead of constant strings. Here is a patch for it. > Any thoughts? A lot of these changes look wrong to me: they are

Re: TRAP: FailedAssertion("HaveRegisteredOrActiveSnapshot()", File: "toast_internals.c", Line: 670, PID: 19403)

2022-04-18 Thread Andres Freund
Hi, On 2022-04-17 11:51:58 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2022-04-16 14:42:39 -0400, Tom Lane wrote: > >> I think (based on the above argument) that what it intends to enforce > >> is not really the system design we need, but it certainly isn't > >> helping anyone that it

Re: Postgres perl module namespace

2022-04-18 Thread Tom Lane
Andres Freund writes: > I just, again, tried to backport a test as part of a bugfix. The > renaming between 14 and 15 makes that task almost comically harder. The > only way I see of dealing with that for the next 5 years is to just > never backpatch tests to < 15. Which seems like a bad outcome.

Re: Postgres perl module namespace

2022-04-18 Thread Andres Freund
Hi, On 2021-10-25 17:12:08 +0900, Michael Paquier wrote: > On Sun, Oct 24, 2021 at 10:46:30AM -0400, Andrew Dunstan wrote: > > ... and pushed. > > Thanks! I just, again, tried to backport a test as part of a bugfix. The renaming between 14 and 15 makes that task almost comically harder. The

Re: A qsort template

2022-04-18 Thread John Naylor
On Tue, Apr 12, 2022 at 7:58 AM David Rowley wrote: > > I've attached the patch I tested. It was thrown together very quickly > just to try out the performance. If it's interesting I can polish it > up a bit. If not, I didn't waste too much time. @@ -959,6 +965,10 @@

Re: pg_walcleaner - new tool to detect, archive and delete the unneeded wal files (was Re: pg_archivecleanup - add the ability to detect, archive and delete the unneeded wal files on the primary)

2022-04-18 Thread Stephen Frost
Greetings, * Bharath Rupireddy (bharath.rupireddyforpostg...@gmail.com) wrote: > Thanks for the comments. Here's a new tool called pg_walcleaner which > basically deletes (optionally archiving before deletion) the unneeded > WAL files. > > Please provide your thoughts and review the patches.

Re: Add --{no-,}bypassrls flags to createuser

2022-04-18 Thread Robert Haas
On Fri, Apr 15, 2022 at 2:33 AM Kyotaro Horiguchi wrote: > > printf(_(" -b, --belongs-to=ROLE new role will be a member of this > > role\n")); > > + printf(_(" -m, --membership=ROLE this role will be a member of > new role\n")); > > membership sounds somewhat obscure, it seems

Re: BufferAlloc: don't take two simultaneous locks

2022-04-18 Thread Robert Haas
On Fri, Apr 15, 2022 at 4:29 AM Kyotaro Horiguchi wrote: > The patch removes buftable entry frist then either inserted again or > returned to freelist. I don't understand how it can be in both > buftable and freelist.. What kind of trouble do you have in mind for > example? I'm not sure. I'm

Re: Re: fix cost subqueryscan wrong parallel cost

2022-04-18 Thread Robert Haas
On Fri, Apr 15, 2022 at 6:06 AM bu...@sohu.com wrote: > > Generally it should be. But there's no subquery scan visible here. > I wrote a patch for distinct/union and aggregate support last year(I want > restart it again). > https://www.postgresql.org/message-id/2021091517250848215321%40sohu.com

GSoC: Database Load Stress Benchmark (2022)

2022-04-18 Thread Mohammad Zain Abbas
Dear concerned, I hope you are doing well. I am Mohammad Zain Abbas, currently enrolled in Erasmus Mundus (BDMA) program. I would like you to have a look at my proposal for the "*Database Load Stress Benchmark" *project. Link:

Replace open mode with PG_BINARY_R/W/A macros

2022-04-18 Thread Japin Li
Hi, hackers I found we defined PG_BINARY_R/W/A macros for opening files, however, there are some places use the constant strings. IMO we should use those macros instead of constant strings. Here is a patch for it. Any thoughts? -- Regrads, Japin Li. ChengDu WenWu Information Technology

Re: TRAP: FailedAssertion("HaveRegisteredOrActiveSnapshot()", File: "toast_internals.c", Line: 670, PID: 19403)

2022-04-18 Thread Robert Haas
On Sat, Apr 16, 2022 at 2:42 PM Tom Lane wrote: > and this blew up with truly impressive thoroughness. The autovac > launcher, logical replication launcher, and incoming backends all > fail this assertion instantly, making it impossible to find out > what else might be broken --- but I'm sure

Re: Column Filtering in Logical Replication

2022-04-18 Thread Tomas Vondra
On 4/18/22 13:04, Amit Kapila wrote: > On Thu, Apr 14, 2022 at 9:09 AM Amit Kapila wrote: >> >> On Thu, Apr 14, 2022 at 8:32 AM Masahiko Sawada >> wrote: The other part of the puzzle is the below check in the code: /* * If we reached the sync worker limit per subscription,

subscribe hackers

2022-04-18 Thread 汪洋
subscribe pgsql-hackers

Re: GSoC: pgagroal: SCRAM-SHA-256-PLUS support (2022)

2022-04-18 Thread Jesper Pedersen
Hi, On 4/18/22 04:16, wanghaitao0...@zju.edu.cn wrote: I'm Haitao Wang, interested in participating in GSOC 2022 PostgreSQL projects. Attached is my proposal. Please check! Thanks for your proposal to Google Summer of Code 2022 ! We'll follow up off-list to get this finalized. Best

Re: Column Filtering in Logical Replication

2022-04-18 Thread Amit Kapila
On Thu, Apr 14, 2022 at 9:09 AM Amit Kapila wrote: > > On Thu, Apr 14, 2022 at 8:32 AM Masahiko Sawada wrote: > > > > > > The other part of the puzzle is the below check in the code: > > > /* > > > * If we reached the sync worker limit per subscription, just exit > > > * silently as we might get

Re: Crash in new pgstats code

2022-04-18 Thread Thomas Munro
On Mon, Apr 18, 2022 at 7:19 PM Michael Paquier wrote: > On Sat, Apr 16, 2022 at 02:36:33PM -0700, Andres Freund wrote: > > which I haven't seen locally. Looks like we have some race between > > startup process and walreceiver? That seems not great. I'm a bit > > confused that walreceiver and

Re: Stabilizing the test_decoding checks, take N

2022-04-18 Thread Dilip Kumar
On Mon, Apr 18, 2022 at 11:19 AM Amit Kapila wrote: > On Sat, Apr 16, 2022 at 10:42 PM Tom Lane wrote: > > > > My pet dinosaur prairiedog just failed in the contrib/test_decoding > > tests [1]: > > > > diff -U3 >

Re: Skipping schema changes in publication

2022-04-18 Thread vignesh C
On Thu, Apr 14, 2022 at 7:18 PM Peter Eisentraut wrote: > > On 12.04.22 08:23, vignesh C wrote: > > I have also included the implementation for skipping a few tables from > > all tables publication, the 0002 patch has the implementation for the > > same. > > This feature is helpful for use cases

Re: pg_receivewal fail to streams when the partial file to write is not fully initialized present in the wal receiver directory

2022-04-18 Thread Bharath Rupireddy
On Tue, Apr 12, 2022 at 5:34 AM Michael Paquier wrote: > > On Mon, Apr 11, 2022 at 01:21:23PM -0700, SATYANARAYANA NARLAPURAM wrote: > > Correct. The idea is to make sure the file is fully allocated before > > treating it as a current file. > > Another problem comes to compression, as the

Re: TRAP: FailedAssertion("tabstat->trans == trans", File: "pgstat_relation.c", Line: 508

2022-04-18 Thread Erikjan Rijkers
Op 18-04-2022 om 09:13 schreef Kyotaro Horiguchi: At Sat, 16 Apr 2022 11:23:23 -0700, Andres Freund wrote in Hi, On 2022-04-16 09:37:55 +0200, Erik Rijkers wrote: I get this crash running the attached test program. On my slow-disked and old desktop it occurs once in 20 or so runs (it is yet

GSoC: pgagroal: SCRAM-SHA-256-PLUS support (2022)

2022-04-18 Thread wanghaitao0125
To whom it may concern: I'm Haitao Wang, interested in participating in GSOC 2022 PostgreSQL projects. Attached is my proposal. Please check! Best Regards, Haitao Wang GSoC pgagroal SCRAM-SHA-256-PLUS support (2022).pdf Description: Adobe PDF document

Re: avoid multiple hard links to same WAL file after a crash

2022-04-18 Thread Michael Paquier
On Fri, Apr 08, 2022 at 09:00:36PM -0400, Robert Haas wrote: > On Fri, Apr 8, 2022 at 12:53 PM Nathan Bossart > wrote: >> I think there might be another problem. The man page for rename() seems to >> indicate that overwriting an existing file also introduces a window where >> the old and new

Fix NULL pointer reference in _outPathTarget()

2022-04-18 Thread Richard Guo
The array sortgrouprefs[] inside PathTarget might be NULL if we have not identified sort/group columns in this tlist. In that case we would have a NULL pointer reference in _outPathTarget() when trying to print sortgrouprefs[] with WRITE_INDEX_ARRAY as we are using the length of PathTarget->exprs

Re: Crash in new pgstats code

2022-04-18 Thread Michael Paquier
On Sat, Apr 16, 2022 at 02:36:33PM -0700, Andres Freund wrote: > which I haven't seen locally. Looks like we have some race between > startup process and walreceiver? That seems not great. I'm a bit > confused that walreceiver and archiving are both active at the same time > in the first place -

Re: TRAP: FailedAssertion("tabstat->trans == trans", File: "pgstat_relation.c", Line: 508

2022-04-18 Thread Kyotaro Horiguchi
At Sat, 16 Apr 2022 11:23:23 -0700, Andres Freund wrote in > Hi, > > On 2022-04-16 09:37:55 +0200, Erik Rijkers wrote: > > I get this crash running the attached test program. On my slow-disked and > > old desktop it occurs once in 20 or so runs (it is yet another installment > > of an old test

Re: Skipping schema changes in publication

2022-04-18 Thread Amit Kapila
On Fri, Apr 15, 2022 at 1:26 AM Euler Taveira wrote: > > On Thu, Apr 14, 2022, at 10:47 AM, Peter Eisentraut wrote: > > On 12.04.22 08:23, vignesh C wrote: > > I have also included the implementation for skipping a few tables from > > all tables publication, the 0002 patch has the implementation

Re: New Object Access Type hooks

2022-04-18 Thread Michael Paquier
On Thu, Mar 24, 2022 at 05:44:31PM -0400, Tom Lane wrote: > Note that that's basically a workaround for buggy placement of the > OAT hooks, as per previous discussion. I hope that we fix that bug > pretty soon, so it shouldn't really be a factor for the meson conversion. So, this issue is still

RE: pg_get_publication_tables() output duplicate relid

2022-04-18 Thread houzj.f...@fujitsu.com
On Friday, April 15, 2022 12:46 AM Alvaro Herrera wrote: > > On 2022-Apr-11, houzj.f...@fujitsu.com wrote: > > > I have confirmed that the bug of ATTACH PARTITION has been fixed due to > recent > > commit 7f481b8. Currently, we always invalidate the RelationSyncCache when > > attaching a

RE: Logical replication timeout problem

2022-04-18 Thread wangw.f...@fujitsu.com
On Thur, Apr 14, 2022 at 8:21 PM Euler Taveira wrote: > Thanks for your comments. > + * For a large transaction, if we don't send any change to the downstream > for a > + * long time then it can timeout. This can happen when all or most of the > + * changes are either not published or got

  1   2   >