Re: Introduce a new view for checkpointer related stats

2022-11-28 Thread Bharath Rupireddy
On Mon, Nov 28, 2022 at 11:29 PM Robert Haas wrote: > > On Tue, Nov 22, 2022 at 3:53 PM Andres Freund wrote: > > I think we should consider deprecating the pg_stat_bgwriter columns but > > leaving them in place for a few years. New stuff should only be added to > > pg_stat_checkpointer, but we

Re: fixing CREATEROLE

2022-11-28 Thread walther
Robert Haas: 1) Automatically install an additional membership grant, with the CREATEROLE user as the grantor, specifying INHERIT OR SET as TRUE (I personally favor attaching these to ALTER ROLE, modifiable only by oneself) Hmm, that's an interesting alternative to what I actually

Re: Decouple last important WAL record LSN from WAL insert locks

2022-11-28 Thread Bharath Rupireddy
On Mon, Nov 28, 2022 at 11:55 PM Andres Freund wrote: > > > Is tracking lastImportantPos as pg_atomic_uint64 in XLogCtlInsert any > > better than an explicit spinlock? I think it's better on platforms > > where atomics are supported, however, it boils down to using a spin > > lock on the

Re: [DOCS] Stats views and functions not in order?

2022-11-28 Thread Peter Smith
On Sat, Nov 26, 2022 at 2:43 PM David G. Johnston wrote: > > On Wed, Nov 23, 2022 at 1:36 AM Peter Smith wrote: >> >> On Thu, Nov 17, 2022 at 8:46 AM David G. Johnston >> wrote: >> >> > Also, make it so each view ends up being its own separate page. >> > >> >> I did not do this. AFAIK those

Re: fixing CREATEROLE

2022-11-28 Thread walther
Robert Haas: And the result is that I've got like five people, some of whom particulated in those discussions, showing up to say "hey, we don't need the ability to set defaults." Well, if that's the case, then why did we have hundreds and hundreds of emails within the last 12 months arguing

Re: Add LZ4 compression in pg_dump

2022-11-28 Thread Michael Paquier
On Tue, Nov 29, 2022 at 03:19:17PM +0900, Michael Paquier wrote: > Attached is an alternative that does not sacrifice the pluggability of > the existing routines while allowing 0003~ to still use them (I don't > really want to move around the checks on the supported build options > now in

Re: fixing CREATEROLE

2022-11-28 Thread walther
Mark Dilger: Isn't this just GRANT .. WITH SET FALSE, INHERIT FALSE, ADMIN TRUE? That should allow role administration, without actually granting membership in that role, yet, right? Can you clarify what you mean here? Are you inventing a new syntax? +GRANT bob TO alice WITH SET FALSE,

RE: Avoid streaming the transaction which are skipped (in corner cases)

2022-11-28 Thread houzj.f...@fujitsu.com
On Tuesday, November 29, 2022 12:08 PM Dilip Kumar wrote: Hi, > > On Mon, Nov 28, 2022 at 3:19 PM Dilip Kumar wrote: > > > > On Mon, Nov 28, 2022 at 1:46 PM shiy.f...@fujitsu.com > > wrote: > > > > > > Thanks for your patch. > > > > > > I saw that the patch added a check when selecting

Re: Collation version tracking for macOS

2022-11-28 Thread Jeff Davis
On Sat, 2022-11-26 at 18:27 +1300, Thomas Munro wrote: > On Thu, Nov 24, 2022 at 5:48 PM Thomas Munro > wrote: > > On Thu, Nov 24, 2022 at 3:07 PM Jeff Davis > > wrote: > > > I'd vote for 1 on the grounds that it's easier to document and > > > understand a single collation version, which comes

Re: Failed Assert while pgstat_unlink_relation

2022-11-28 Thread Kyotaro Horiguchi
At Mon, 28 Nov 2022 14:01:30 +0530, vignesh C wrote in > Hi, > > While reviewing/testing one of the patches I found the following Assert: > #0 0x55c6312ba639 in pgstat_unlink_relation (rel=0x7fb73bcbac58) > at pgstat_relation.c:161 > #1 0x55c6312bbb5a in

Re: Support logical replication of DDLs

2022-11-28 Thread Ajin Cherian
On Tue, Nov 29, 2022 at 4:22 PM rajesh singarapu wrote: > > Isn't it a good idea to move triggers to CreateReplicationSlot() ? as > create publication also create replication slot, AFAIK. > > thanks > Rajesh > Currently we're trying to get this work using "Create Publication", maybe in future

Re: Add LZ4 compression in pg_dump

2022-11-28 Thread Michael Paquier
On Mon, Nov 28, 2022 at 04:32:43PM +, gkokola...@pm.me wrote: > The focus of this version of this series is 0001 and 0002. > > Admittedly 0001 could be presented in a separate thread though given its size > and > proximity to the topic, I present it here. I don't mind. This was a hole in

Re: Fix comment in SnapBuildFindSnapshot

2022-11-28 Thread Masahiko Sawada
On Tue, Nov 29, 2022 at 8:54 AM Michael Paquier wrote: > > On Mon, Nov 28, 2022 at 04:46:44PM +0900, Michael Paquier wrote: > > Hm, yes, that seems right. There are three "c) states" in these > > paragraphs, they are incremental steps. Will apply if there are no > > objections. > > And done.

Re: Support logical replication of DDLs

2022-11-28 Thread rajesh singarapu
Isn't it a good idea to move triggers to CreateReplicationSlot() ? as create publication also create replication slot, AFAIK. thanks Rajesh On Tue, Nov 29, 2022 at 10:35 AM rajesh singarapu wrote: > > Thanks Ajin for the clarification. > > This is really a helpful feature. > > On Tue, Nov 29,

RE: Perform streaming logical transactions by background workers and parallel apply

2022-11-28 Thread houzj.f...@fujitsu.com
On Mon, November 28, 2022 15:19 PM Peter Smith wrote: > Here are some review comments for patch v51-0002 Thanks for your comments! > == > > 1. > > GENERAL - terminology: spool/serialize and data/changes/message > > The terminology seems to be used at random. IMO it might be worthwhile

Re: Support logical replication of DDLs

2022-11-28 Thread rajesh singarapu
Thanks Ajin for the clarification. This is really a helpful feature. On Tue, Nov 29, 2022 at 10:14 AM Ajin Cherian wrote: > > On Tue, Nov 29, 2022 at 3:39 PM rajesh singarapu > wrote: > > > > Thanks Ajin for the reply. > > > > We "Create/Install" these trigger function at the time of "Create >

Re: Collation version tracking for macOS

2022-11-28 Thread Jeff Davis
On Mon, 2022-11-28 at 14:11 -0500, Robert Haas wrote: > I don't really understand #1 or #5 well enough to have an educated > opinion, but I do think that #1 seems a bit magical. It hopes that > the > combination of a collation name and a datcollversion will be > sufficient to find exactly one

Re: Support logical replication of DDLs

2022-11-28 Thread Ajin Cherian
On Tue, Nov 29, 2022 at 3:39 PM rajesh singarapu wrote: > > Thanks Ajin for the reply. > > We "Create/Install" these trigger function at the time of "Create > publication", (CreatePublication()) > but If I create a replication slot using something like "select * from >

Re: Support logical replication of DDLs

2022-11-28 Thread rajesh singarapu
Thanks Ajin for the reply. We "Create/Install" these trigger function at the time of "Create publication", (CreatePublication()) but If I create a replication slot using something like "select * from pg_create_logical_replication_slot('test1', 'test_decoding')" we would not install these triggers

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-11-28 Thread John Naylor
> The fix is easy enough -- set the child pointer to null upon deletion, but I'm somewhat astonished that the regression tests didn't hit this. I do still intend to replace this code with something faster, but before I do so the tests should probably exercise the deletion paths more. Since VACUUM

Re: [PoC] Improve dead tuple storage for lazy vacuum

2022-11-28 Thread John Naylor
While creating a benchmark for inserting into node128-inner, I found a bug. If a caller deletes from a node128, the slot index is set to invalid, but the child pointer is still valid. Do that a few times, and every child pointer is valid, even if no slot index points to it. When the next inserter

Re: Collation version tracking for macOS

2022-11-28 Thread Thomas Munro
On Tue, Nov 29, 2022 at 3:55 PM Jeff Davis wrote: > =# select * from pg_icu_collation_versions('en_US') order by > icu_version; > icu_version | uca_version | collator_version > -+-+-- > 50.2| 6.2 | 58.0.6.50 > 51.3| 6.2 |

Re: Support logical replication of DDLs

2022-11-28 Thread Ajin Cherian
On Tue, Nov 29, 2022 at 1:29 PM rajesh singarapu wrote: > > One question, > > I understand that we create/enable triggers on create publication command > flow. > I am wondering how this works in case of logical replication using slots. > > > thanks > Rajesh > Rajesh, The triggers functions when

Re: Avoid streaming the transaction which are skipped (in corner cases)

2022-11-28 Thread Dilip Kumar
On Mon, Nov 28, 2022 at 3:19 PM Dilip Kumar wrote: > > On Mon, Nov 28, 2022 at 1:46 PM shiy.f...@fujitsu.com > wrote: > > > > Thanks for your patch. > > > > I saw that the patch added a check when selecting largest transaction, but > > in > > addition to ReorderBufferCheckMemoryLimit(), the

Re: Collation version tracking for macOS

2022-11-28 Thread Jeff Davis
On Mon, 2022-11-28 at 21:57 -0500, Robert Haas wrote: > That is ... astonishingly bad. https://unicode-org.atlassian.net/browse/CLDR-16175 -- Jeff Davis PostgreSQL Contributor Team - AWS

Re: Collation version tracking for macOS

2022-11-28 Thread Robert Haas
On Mon, Nov 28, 2022 at 9:55 PM Jeff Davis wrote: > But did you notice that the version went backwards from 65.1 -> 66.1? > Well, actually, it didn't. The version of that collation in 66.1 went > from 153.97 -> 153.104. But there's a bug in versionToString() that > does the decimal output

Re: Collation version tracking for macOS

2022-11-28 Thread Jeff Davis
On Sat, 2022-11-26 at 18:27 +1300, Thomas Munro wrote: > Here's the first iteration. I will send a full review shortly, but I encountered an ICU bug along the way, which caused me some confusion for a bit. I'll skip past the various levels of confusion I had (burned a couple hours), and get right

Re: PGDOCS - Logical replication GUCs - added some xrefs

2022-11-28 Thread Peter Smith
On Fri, Nov 25, 2022 at 9:23 PM Peter Eisentraut wrote: > > Your patch moves the description of the subscriber-related configuration > parameters from config.sgml to logical-replication.sgml. But > config.sgml is supposed to contain *all* configuration parameters. If > we're going to start

Re: Support logical replication of DDLs

2022-11-28 Thread rajesh singarapu
One question, I understand that we create/enable triggers on create publication command flow. I am wondering how this works in case of logical replication using slots. thanks Rajesh On Mon, Nov 28, 2022 at 10:17 AM Zheng Li wrote: > > > > > > > > > 1. CREATE TABLE LIKE > > > > I found that

Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)

2022-11-28 Thread Melanie Plageman
On Wed, Nov 23, 2022 at 12:43 AM Justin Pryzby wrote: > > Note that 001 fails to compile without 002: > > ../src/backend/storage/buffer/bufmgr.c:1257:43: error: ‘from_ring’ undeclared > (first use in this function) > 1257 | StrategyRejectBuffer(strategy, buf, from_ring)) Thanks! I fixed

Re: fixing CREATEROLE

2022-11-28 Thread Robert Haas
On Mon, Nov 28, 2022 at 6:32 PM David G. Johnston wrote: > Desirable follow-on patches include: > > 1) Automatically install an additional membership grant, with the CREATEROLE > user as the grantor, specifying INHERIT OR SET as TRUE (I personally favor > attaching these to ALTER ROLE,

Re: Preventing indirection for IndexPageGetOpaque for known-size page special areas

2022-11-28 Thread Tom Lane
Michael Paquier writes: > I am wondering if it would be worth adding an AssertMacro() like in > this one, though: > https://www.postgresql.org/message-id/ykap64jvztmgc...@paquier.xyz Kind of doubt it. It'd bloat debug builds with a lot of redundant checks, and probably never catch anything.

Re: Preventing indirection for IndexPageGetOpaque for known-size page special areas

2022-11-28 Thread Michael Paquier
On Mon, Nov 28, 2022 at 05:31:50PM -0500, Tom Lane wrote: > After looking through this thread, I side with Robert: we should reject > the remainder of this patch. It gives up page layout flexibility that > we might want back someday. Moreover, I didn't see any hard evidence > offered that

Re: Add index scan progress to pg_stat_progress_vacuum

2022-11-28 Thread Imseih (AWS), Sami
> I think that indexes_total should be 0 also when INDEX_CLEANUP is off. Patch updated for handling of INDEX_CLEANUP = off, with an update to the documentation as well. >I think we don't need to reset it at the end of index vacuuming. There >is a small window before switching to the

Re: Fix comment in SnapBuildFindSnapshot

2022-11-28 Thread Michael Paquier
On Mon, Nov 28, 2022 at 04:46:44PM +0900, Michael Paquier wrote: > Hm, yes, that seems right. There are three "c) states" in these > paragraphs, they are incremental steps. Will apply if there are no > objections. And done. -- Michael signature.asc Description: PGP signature

Re: Allow file inclusion in pg_hba and pg_ident files

2022-11-28 Thread Michael Paquier
On Mon, Nov 28, 2022 at 04:12:40PM +0900, Michael Paquier wrote: > Attached is a rebased patch of the rest. With everything we have > dealt with in this CF, perhaps it would be better to mark this entry > as committed and switch to a new thread where the negative TAP tests > could be discussed?

Re: Non-decimal integer literals

2022-11-28 Thread David Rowley
On Sat, 26 Nov 2022 at 05:13, Peter Eisentraut wrote: > > On 24.11.22 10:13, David Rowley wrote: > > I > > remember many years ago and several jobs ago when working with SQL > > Server being able to speed up importing data using hexadecimal > > DATETIMEs. I can't think why else you might want to

Re: fixing CREATEROLE

2022-11-28 Thread Robert Haas
On Mon, Nov 28, 2022 at 4:55 PM Robert Haas wrote: > But so far nobody has actually reviewed anything, ... Actually this isn't true. Mark did review. Thanks, Mark. -- Robert Haas EDB: http://www.enterprisedb.com

Re: O(n) tasks cause lengthy startups and checkpoints

2022-11-28 Thread Nathan Bossart
Okay, here is a new patch set. 0004 adds logic to prevent custodian tasks from delaying shutdown. I haven't added any logging for long-running tasks yet. Tasks might ordinarily take a while, so such logs wouldn't necessarily indicate something is wrong. Perhaps we could add a GUC for the

Re: fixing CREATEROLE

2022-11-28 Thread David G. Johnston
On Mon, Nov 28, 2022 at 2:55 PM Robert Haas wrote: > On Mon, Nov 28, 2022 at 4:19 PM David G. Johnston > wrote: > > That's fine, but are you saying this patch is incapable (or simply > undesirable) of having the parts about handling defaults separated out from > the parts that define how the

Re: Report roles in pg_upgrade pg_ prefix check

2022-11-28 Thread Michael Paquier
On Mon, Nov 28, 2022 at 09:58:46AM +0100, Daniel Gustafsson wrote: > We are a bit inconsistent in how much details we include in the report > textfiles, so could do that without breaking any consistency in reporting. > Looking at other checks, the below format would match what we already do >

Re: Reducing power consumption on idle servers

2022-11-28 Thread Thomas Munro
I found some more comments and some documentation to word-smith very lightly, and pushed. The comments were stray references to the trigger file. It's a little confusing because the remaining mechanism also uses a file, but it uses a signal first so seems better to refer to promotion requests

Re: Add 64-bit XIDs into PostgreSQL 15

2022-11-28 Thread Peter Geoghegan
On Mon, Nov 28, 2022 at 1:52 PM Peter Geoghegan wrote: > I'm not claiming to know how to build this "better xidStopLimit > mechanism", by the way. I'm not seriously proposing it. Mostly I'm > just saying that the question "where do you draw the line if not at 2 > billion XIDs?" is a very

Re: Preventing indirection for IndexPageGetOpaque for known-size page special areas

2022-11-28 Thread Tom Lane
Bruce Momjian writes: > Uh, XTS doesn't require a nonce, so why are talking about nonces in this > thread? Because some other proposals do, or could, require a per-page nonce. After looking through this thread, I side with Robert: we should reject the remainder of this patch. It gives up page

Re: [PATCH] Infinite loop while acquiring new TOAST Oid

2022-11-28 Thread Nikita Malakhov
I've missed that - >4 billion external datums >typically use a lot of space. Not quite so. It's 8 Tb for the minimal size of toasted data (about 2 Kb). In my practice tables with more than 5 billions of rows are not of something out of the ordinary (highly loaded databases with large amounts of

Re: [PATCH] Infinite loop while acquiring new TOAST Oid

2022-11-28 Thread Tom Lane
Andres Freund writes: > On 2022-11-28 16:57:53 -0500, Tom Lane wrote: >> As I said before, I think there's a decent argument that some people >> will want the option to stay with 4-byte TOAST OIDs indefinitely, >> at least for smaller tables. > And as you said earlier, the increased overhead

Re: [PATCH] Infinite loop while acquiring new TOAST Oid

2022-11-28 Thread Nikita Malakhov
Hi, I'll check that tomorrow. If it is so then there won't be a problem keeping old tables without re-toasting. On Tue, Nov 29, 2022 at 1:10 AM Andres Freund wrote: > Hi, > > On 2022-11-28 16:57:53 -0500, Tom Lane wrote: > > As I said before, I think there's a decent argument that some people

Re: [PATCH] Infinite loop while acquiring new TOAST Oid

2022-11-28 Thread Andres Freund
Hi, On 2022-11-28 16:57:53 -0500, Tom Lane wrote: > As I said before, I think there's a decent argument that some people > will want the option to stay with 4-byte TOAST OIDs indefinitely, > at least for smaller tables. I think we'll need to do something about the width of varatt_external to

Re: Add 64-bit XIDs into PostgreSQL 15

2022-11-28 Thread Peter Geoghegan
On Mon, Nov 28, 2022 at 1:52 PM Bruce Momjian wrote: > I think the problem is that we still have bloat with 64-bit XIDs, > specifically pg_xact and pg_multixact files. Yes, that bloat is less > serious, but it is still an issue worth reporting in the server logs, > though not serious enough to

Re: [PATCH] Infinite loop while acquiring new TOAST Oid

2022-11-28 Thread Tom Lane
Andres Freund writes: > On 2022-11-29 00:24:49 +0300, Nikita Malakhov wrote: >> 2) Upgrading to 64-bit id would require re-toasting old TOAST tables. Or >> there is some way to distinguish old tables from new ones? > The catalog / relcache entry should suffice to differentiate between the two.

Re: fixing CREATEROLE

2022-11-28 Thread Robert Haas
On Mon, Nov 28, 2022 at 4:19 PM David G. Johnston wrote: > That's fine, but are you saying this patch is incapable (or simply > undesirable) of having the parts about handling defaults separated out from > the parts that define how the system works with a given set of permissions; > and the

Re: An attempt to avoid locally-committed-but-not-replicated-to-standby-transactions in synchronous replication

2022-11-28 Thread Bruce Momjian
On Mon, Nov 28, 2022 at 01:31:39PM -0800, Andrey Borodin wrote: > On Mon, Nov 28, 2022 at 12:59 PM Bruce Momjian wrote: > > > > You can prepare a patch, but it unlikely to get much interest until you > > get agreement on what the behavior should be. > > We discussed the approach on 2020's

Re: Add 64-bit XIDs into PostgreSQL 15

2022-11-28 Thread Peter Geoghegan
On Mon, Nov 28, 2022 at 1:30 PM Robert Haas wrote: > What is the purpose of using 64-bit XIDs, if not to avoid having to > stop the world when we run short of XIDs? I agree that the xidStopLimit mechanism was designed with the specific goal of preventing "true" physical XID wraparound that

Re: Add 64-bit XIDs into PostgreSQL 15

2022-11-28 Thread Bruce Momjian
On Mon, Nov 28, 2022 at 04:30:22PM -0500, Robert Haas wrote: > What is the purpose of using 64-bit XIDs, if not to avoid having to > stop the world when we run short of XIDs? > > I'd say that if this patch, or any patch with broadly similar goals, > fails to remove xidStopLimit, it might as well

Re: [PATCH] Infinite loop while acquiring new TOAST Oid

2022-11-28 Thread Andres Freund
Hi, On 2022-11-29 00:24:49 +0300, Nikita Malakhov wrote: > 2) Upgrading to 64-bit id would require re-toasting old TOAST tables. Or > there is some way to distinguish old tables from new ones? The catalog / relcache entry should suffice to differentiate between the two. Greetings, Andres

Re: [PATCH] Infinite loop while acquiring new TOAST Oid

2022-11-28 Thread Tom Lane
Andres Freund writes: > On 2022-11-28 16:04:12 -0500, Tom Lane wrote: >> And I don't buy that either. An extra 4 bytes with a 2K payload is not >> "prohibitive", it's more like "down in the noise". > The space usage for the the the toast relation + index itself is indeed > irrelevant. Where

Re: CF 2022-11: entries "Ready for Committer" with recent activity

2022-11-28 Thread Bruce Momjian
On Sun, Nov 27, 2022 at 01:29:18PM +0900, Ian Lawrence Barwick wrote: > Transaction Management docs (2022-11-23) > > > - https://commitfest.postgresql.org/40/3899/ > - >

Re: Failed Assert in pgstat_assoc_relation

2022-11-28 Thread Tom Lane
Andres Freund writes: > Something like the attached. Still needs a bit of polish, e.g. adding the test > case from above. > I'm a bit uncomfortable adding a function call below >* Perform swapping of the relcache entry contents. Within this >* process the old

Re: An attempt to avoid locally-committed-but-not-replicated-to-standby-transactions in synchronous replication

2022-11-28 Thread Andrey Borodin
On Mon, Nov 28, 2022 at 12:59 PM Bruce Momjian wrote: > > You can prepare a patch, but it unlikely to get much interest until you > get agreement on what the behavior should be. We discussed the approach on 2020's Unconference [0]. And there kind of was an agreement. Then I made a presentation

Re: Add 64-bit XIDs into PostgreSQL 15

2022-11-28 Thread Robert Haas
On Mon, Nov 28, 2022 at 4:09 PM Peter Geoghegan wrote: > Granted, the specifics of the current XidStopLimit mechanism are > unlikely to directly carry over to 64-bit XIDs. XidStopLimit is > structured in a way that doesn't actually consider freeze debt in > units like unfrozen pages. Like Chris,

Re: Slow standby snapshot

2022-11-28 Thread Tom Lane
Michail Nikolaev writes: >> * when idle - since we have time to do it when that happens, which >> happens often since most workloads are bursty > I have added getting of ProcArrayLock for this case. That seems like a fairly bad idea: it will add extra contention on ProcArrayLock, and I see no

Re: Patch: Global Unique Index

2022-11-28 Thread Bruce Momjian
On Fri, Nov 25, 2022 at 05:03:06PM -0800, David Zhang wrote: > Hi Bruce, > > Thank you for helping review the patches in such detail. > > On 2022-11-25 9:48 a.m., Bruce Momjian wrote: > > Looking at the patch, I am unclear how the the patch prevents concurrent > duplicate value

Re: [PATCH] Infinite loop while acquiring new TOAST Oid

2022-11-28 Thread Nikita Malakhov
Hi, Andres Freund writes: >Was the issue that you exceeded 4 billion toasted datums, or that assignment >took a long time? How many toast datums did you actually have? Was this due to >very wide rows leading to even small datums getting toasted? Yep, we had 4 billion toasted datums. I remind

Re: [PATCH] Infinite loop while acquiring new TOAST Oid

2022-11-28 Thread Andres Freund
Hi, On 2022-11-28 16:04:12 -0500, Tom Lane wrote: > Andres Freund writes: > > - to > > combat the space usage we likely should switch to representing the ids as > > variable width integers or such, otherwise the space increase would likely > > be > > prohibitive. > > And I don't buy that

Re: fixing CREATEROLE

2022-11-28 Thread David G. Johnston
On Mon, Nov 28, 2022 at 1:28 PM Robert Haas wrote: > On Mon, Nov 28, 2022 at 3:02 PM Mark Dilger > wrote: > > You can argue that a grant with INHERIT FALSE, SET FALSE, ADMIN TRUE > still grants membership, and I think formally that's true, but I also > think it's just picking something to

Re: Bug in wait time when waiting on nested subtransaction

2022-11-28 Thread Robert Haas
On Mon, Nov 28, 2022 at 3:01 PM Tom Lane wrote: > One thing we need to be pretty careful of here is to not break the > promise of atomic commit. At topmost commit, all subxacts must > appear committed simultaneously. It's not quite clear to me whether > we need a similar guarantee in the

Re: Add 64-bit XIDs into PostgreSQL 15

2022-11-28 Thread Peter Geoghegan
On Mon, Nov 28, 2022 at 8:53 AM Robert Haas wrote: > It is true that if the table is progressively bloating, it is likely > to be more bloated by the time you are 8 billion XIDs behind than it > was when you were 800 million XIDs behind. I don't see that as a very > good reason not to adopt this

Re: Failed Assert in pgstat_assoc_relation

2022-11-28 Thread Andres Freund
Hi, On 2022-11-28 10:50:13 -0800, Andres Freund wrote: > On 2022-11-28 13:37:16 -0500, Tom Lane wrote: > > Uh-huh. I've not bothered to trace this in detail, but presumably > > what is happening is that the first CREATE RULE converts the table > > to a view, and then the ROLLBACK undoes that so

Re: [PATCH] Infinite loop while acquiring new TOAST Oid

2022-11-28 Thread Tom Lane
Andres Freund writes: > I think the first step to improve the situation is to not use a global oid > counter for toasted values. One way to do that would be to use the sequence > code to do oid assignment, but we likely can find a more efficient > representation. I don't particularly buy that,

Re: [PATCH] Infinite loop while acquiring new TOAST Oid

2022-11-28 Thread Andres Freund
Hi, On 2022-11-28 23:54:53 +0300, Nikita Malakhov wrote: > We've already encountered this issue on large production databases, and > 4 billion rows is not so much for modern bases, so this issue already arises > from time to time and would arise more and more often. Was the issue that you

Re: An attempt to avoid locally-committed-but-not-replicated-to-standby-transactions in synchronous replication

2022-11-28 Thread Bruce Momjian
On Mon, Nov 28, 2022 at 12:03:06PM +0530, Bharath Rupireddy wrote: > Thanks for verifying the behaviour. And many thanks for an off-list chat. > > FWIW, I'm planning to prepare a patch as per the below idea which is > something similar to the initial proposal in this thread. Meanwhile, > thoughts

Re: An attempt to avoid locally-committed-but-not-replicated-to-standby-transactions in synchronous replication

2022-11-28 Thread Bruce Momjian
On Sun, Nov 27, 2022 at 11:26:50AM -0800, Andrey Borodin wrote: > Some funny stuff. If a user tries to cancel a non-replicated transaction > Azure Postgres will answer: "user requested cancel while waiting for > synchronous replication ack. The COMMIT record has already flushed to > WAL locally

Re: [PATCH] Infinite loop while acquiring new TOAST Oid

2022-11-28 Thread Nikita Malakhov
Hi! We've already encountered this issue on large production databases, and 4 billion rows is not so much for modern bases, so this issue already arises from time to time and would arise more and more often. I agree that global oid counter is the main issue, and better solution would be local

Re: fixing CREATEROLE

2022-11-28 Thread Mark Dilger
> On Nov 28, 2022, at 12:33 PM, Mark Dilger > wrote: > >> Isn't this just GRANT .. WITH SET FALSE, INHERIT FALSE, ADMIN TRUE? That >> should allow role administration, without actually granting membership in >> that role, yet, right? > > Can you clarify what you mean here? Are you

Re: [PATCH] Infinite loop while acquiring new TOAST Oid

2022-11-28 Thread Andres Freund
Hi, On 2022-11-28 18:34:20 +0300, Nikita Malakhov wrote: > While working on Pluggable TOAST we've detected a defective behavior > on tables with large amounts of TOASTed data - queries freeze and DB stalls. > Further investigation led us to the loop with GetNewOidWithIndex function > call - when

Re: fixing CREATEROLE

2022-11-28 Thread Mark Dilger
> On Nov 28, 2022, at 12:08 PM, walt...@technowledgy.de wrote: > > Isn't this just GRANT .. WITH SET FALSE, INHERIT FALSE, ADMIN TRUE? That > should allow role administration, without actually granting membership in > that role, yet, right? Can you clarify what you mean here? Are you

Re: fixing CREATEROLE

2022-11-28 Thread Robert Haas
On Mon, Nov 28, 2022 at 3:02 PM Mark Dilger wrote: > Robert's patch tries to deal with the (possibly unwanted) role membership by > setting up defaults to mitigate the effects, but that is more confusing to me > than just de-conflating role membership from role administration, and giving >

Re: fixing CREATEROLE

2022-11-28 Thread walther
Mark Dilger: Robert's patch tries to deal with the (possibly unwanted) role membership by setting up defaults to mitigate the effects, but that is more confusing to me than just de-conflating role membership from role administration, and giving role creators administration over roles they

Re: fixing CREATEROLE

2022-11-28 Thread Mark Dilger
> On Nov 28, 2022, at 11:34 AM, David G. Johnston > wrote: > > No Defaults needed: David J., Mark?, Tom? As Robert has the patch organized, I think defaults are needed, but I see that as a strike against the patch. > Defaults needed - attached to role directly: Robert > Defaults needed -

Re: Bug in wait time when waiting on nested subtransaction

2022-11-28 Thread Tom Lane
Robert Haas writes: > That's not what "running" means to me. Running means it's started and > hasn't yet committed or rolled back. A subxact definitely can't be considered committed until its topmost parent commits. However, it could be known to be rolled back before its parent. IIUC, the

Re: Failed Assert in pgstat_assoc_relation

2022-11-28 Thread Tom Lane
Andres Freund writes: > On 2022-11-28 13:37:16 -0500, Tom Lane wrote: >> As far as HEAD is concerned, maybe it's time to nuke the whole >> convert-table-to-view kluge entirely? Only pg_dump older than >> 9.4 will emit such code, so we're really about out of reasons >> to keep on maintaining it.

Re: fixing CREATEROLE

2022-11-28 Thread walther
Robert Haas: In my proposal, the "object" is not the GRANT of that role. It's the role itself. So the default privileges express what should happen when the role is created. The default privileges would NOT affect a regular GRANT role TO role_spec command. They only run that command when a role

Re: fixing CREATEROLE

2022-11-28 Thread David G. Johnston
On Mon, Nov 28, 2022 at 12:42 PM wrote: > David G. Johnston: > > A quick tally of the thread so far: > > > > No Defaults needed: David J., Mark?, Tom? > > Defaults needed - attached to role directly: Robert > > Defaults needed - defined within Default Privileges: Walther? > > s/Walther/Wolfgang

Re: Bug in wait time when waiting on nested subtransaction

2022-11-28 Thread Robert Haas
On Mon, Nov 28, 2022 at 2:45 PM Simon Riggs wrote: > An easy point to confuse: > "subtransaction to end": The subtransaction is "still running" to > other backends even AFTER it has been subcommitted, but its state now > transfers to the parent. > > So the subtransaction doesn't cease running

Re: Bug in wait time when waiting on nested subtransaction

2022-11-28 Thread Simon Riggs
On Mon, 28 Nov 2022 at 17:38, Robert Haas wrote: > > On Mon, Nov 28, 2022 at 10:28 AM Simon Riggs > wrote: > > So we have these options > > > > 1. Removing the XactLockTableDelete() call in CommitSubTransaction(). > > That releases lock waiters earlier than expected, which requires > > pushups

Re: fixing CREATEROLE

2022-11-28 Thread walther
David G. Johnston: A quick tally of the thread so far: No Defaults needed: David J., Mark?, Tom? Defaults needed - attached to role directly: Robert Defaults needed - defined within Default Privileges: Walther? s/Walther/Wolfgang The capability itself seems orthogonal to the rest of the

Re: fixing CREATEROLE

2022-11-28 Thread Robert Haas
On Mon, Nov 28, 2022 at 1:56 PM wrote: > And now this reason is gone - there is no reason NOT to implement it as > DEFAULT PRIVILEGES. I think there is, and it's this, which you wrote further down: > In my proposal, the "object" is not the GRANT of that role. It's the > role itself. So the

Re: fixing CREATEROLE

2022-11-28 Thread David G. Johnston
On Mon, Nov 28, 2022 at 11:57 AM wrote: > Robert Haas: > > I don't know if changing the syntax from A to B is really getting us > > anywhere. I generally agree that the ALTER DEFAULT PRIVILEGES syntax > > looks nicer than the CREATE/ALTER ROLE syntax, but I'm not sure that's > > a sufficient

Re: Bug in wait time when waiting on nested subtransaction

2022-11-28 Thread Simon Riggs
On Mon, 28 Nov 2022 at 18:53, Alvaro Herrera wrote: > > On 2022-Nov-28, Simon Riggs wrote: > > > A narrative description of the issue follows: > > session1 - requests multiple nested subtransactions like this: > > BEGIN; ... > > SAVEPOINT subxid1; ... > > SAVEPOINT subxid2; ... > > > However, if

Re: TAP output format in pg_regress

2022-11-28 Thread Daniel Gustafsson
> On 28 Nov 2022, at 20:02, Nikolay Shaplov wrote: > From my reviewer's point of view patch is ready for commit. > > Thank you for your patience :-) Thanks for review. The attached tweaks a few comments and attempts to address the compiler warning error in the CFBot CI. Not sure I entirely

Re: Collation version tracking for macOS

2022-11-28 Thread Robert Haas
On Wed, Nov 23, 2022 at 12:09 AM Thomas Munro wrote: > OK. Time for a new list of the various models we've discussed so far: > > 1. search-by-collversion: We introduce no new "library version" > concept to COLLATION and DATABASE object and little or no new syntax. > > 2.

Re: TAP output format in pg_regress

2022-11-28 Thread Nikolay Shaplov
В письме от понедельник, 28 ноября 2022 г. 21:28:48 MSK пользователь Andres Freund написал: > On 2022-11-28 14:13:16 +0100, Daniel Gustafsson wrote: > > > On 27 Nov 2022, at 11:22, Nikolay Shaplov wrote: > > > В письме от суббота, 26 ноября 2022 г. 23:35:45 MSK пользователь Daniel > > >

Re: Add tracking of backend memory allocated to pg_stat_activity

2022-11-28 Thread Andres Freund
On 2022-11-26 22:10:06 -0500, Reid Thompson wrote: >- zero allocated bytes after fork to avoid double counting postmaster > allocations I still don't understand this - postmaster shouldn't be counted at all. It doesn't have a PgBackendStatus. There simply shouldn't be any tracked allocations

Re: fixing CREATEROLE

2022-11-28 Thread walther
Robert Haas: I don't know if changing the syntax from A to B is really getting us anywhere. I generally agree that the ALTER DEFAULT PRIVILEGES syntax looks nicer than the CREATE/ALTER ROLE syntax, but I'm not sure that's a sufficient reason to move the control over this behavior to ALTER

Re: Bug in wait time when waiting on nested subtransaction

2022-11-28 Thread Alvaro Herrera
On 2022-Nov-28, Simon Riggs wrote: > A narrative description of the issue follows: > session1 - requests multiple nested subtransactions like this: > BEGIN; ... > SAVEPOINT subxid1; ... > SAVEPOINT subxid2; ... > However, if subxid2 subcommits, then the lock wait moves from subxid2 > to the

Re: predefined role(s) for VACUUM and ANALYZE

2022-11-28 Thread Nathan Bossart
On Mon, Nov 28, 2022 at 12:13:13PM -0500, Andrew Dunstan wrote: > pushed. Thanks! -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: Another multi-row VALUES bug

2022-11-28 Thread Tom Lane
Dean Rasheed writes: > A different way to do this, without relying on the contents of the > targetlist, is to note that, while processing a product query, what we > really want to do is ignore any VALUES RTEs from the original query, > since they will have already been processed. There should

Re: Failed Assert in pgstat_assoc_relation

2022-11-28 Thread Andres Freund
Hi, On 2022-11-28 13:37:16 -0500, Tom Lane wrote: > vignesh C writes: > > I could reproduce this issue with the following steps: > > create table t1(c int); > > BEGIN; > > CREATE TABLE t (c int); > > SAVEPOINT q; > > CREATE RULE "_RETURN" AS ON SELECT TO t DO INSTEAD SELECT * FROM t1; > >

Re: Failed Assert in pgstat_assoc_relation

2022-11-28 Thread Tom Lane
vignesh C writes: > I could reproduce this issue with the following steps: > create table t1(c int); > BEGIN; > CREATE TABLE t (c int); > SAVEPOINT q; > CREATE RULE "_RETURN" AS ON SELECT TO t DO INSTEAD SELECT * FROM t1; > select * from t; > ROLLBACK TO q; > CREATE RULE "_RETURN" AS ON SELECT

Re: O(n) tasks cause lengthy startups and checkpoints

2022-11-28 Thread Robert Haas
On Mon, Nov 28, 2022 at 1:31 PM Andres Freund wrote: > On 2022-11-28 13:08:57 +, Simon Riggs wrote: > > On Sun, 27 Nov 2022 at 23:34, Nathan Bossart > > wrote: > > > > Rather than explicitly use DEBUG1 everywhere I would have an > > > > #define CUSTODIAN_LOG_LEVEL LOG > > > > so we can

  1   2   >