Re: Adding a LogicalRepWorker type field

2023-08-11 Thread Amit Kapila
On Sat, Aug 12, 2023 at 5:01 AM Peter Smith wrote: > > On Fri, Aug 11, 2023 at 9:13 PM Amit Kapila wrote: > > > > On Fri, Aug 11, 2023 at 3:41 PM Peter Smith wrote: > > > > > > On Fri, Aug 11, 2023 at 7:33 PM Amit Kapila > > > wrote: > > > > > > > > On Thu, Aug 10, 2023 at 7:50 AM Peter Smith

Re: A failure in 031_recovery_conflict.pl on Debian/s390x

2023-08-11 Thread Thomas Munro
Thanks. I realised that it's easy enough to test that theory about cleanup locks by hacking ConditionalLockBufferForCleanup() to return false randomly. Then the test occasionally fails as described. Seems like we'll need to fix that test, but it's not evidence of a server bug, and my signal

CREATE FUNCTION ... SEARCH { DEFAULT | SYSTEM | SESSION }

2023-08-11 Thread Jeff Davis
The attached patch implements a new SEARCH clause for CREATE FUNCTION. The SEARCH clause controls the search_path used when executing functions that were created without a SET clause. Background: Controlling search_path is critical for the correctness and security of functions. Right now, the

Re: Schema variables - new implementation for Postgres 15

2023-08-11 Thread Julien Rouhaud
On Fri, Aug 11, 2023 at 05:55:26PM +0200, Dmitry Dolgov wrote: > > Another confusing example was this one at the end of set_session_variable: > > + /* > + * XXX While unlikely, an error here is possible. It wouldn't leak > memory > + * as the allocated chunk has already been

Re: run pgindent on a regular basis / scripted manner

2023-08-11 Thread Andres Freund
Hi, On 2023-08-11 20:11:34 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2023-08-11 18:30:02 -0400, Tom Lane wrote: > >> +1 for including this in CI tests > > > I didn't even mean CI - I meant 'make check-world' / 'meson test'. Which of > > course would include CI automatically. > > Hmm.

Rename ExtendedBufferWhat in 16?

2023-08-11 Thread Thomas Munro
Hi Commit 31966b15 invented a way for functions dealing with relation extension to accept a Relation in online code and an SMgrRelation in recovery code (instead of using the earlier FakeRelcacheEntry concept). It seems highly likely that future new bufmgr.c interfaces will face the same

Re: run pgindent on a regular basis / scripted manner

2023-08-11 Thread Michael Paquier
On Fri, Aug 11, 2023 at 08:11:34PM -0400, Tom Lane wrote: > Hmm. I'm allergic to anything that significantly increases the cost > of check-world, and this seems like it'd do that. > > Maybe we could automate it, but not as part of check-world per se? It does not have to be part of check-world

Re: run pgindent on a regular basis / scripted manner

2023-08-11 Thread Tom Lane
Andres Freund writes: > On 2023-08-11 18:30:02 -0400, Tom Lane wrote: >> +1 for including this in CI tests > I didn't even mean CI - I meant 'make check-world' / 'meson test'. Which of > course would include CI automatically. Hmm. I'm allergic to anything that significantly increases the cost

Re: logicalrep_worker_launch -- counting/checking the worker limits

2023-08-11 Thread Peter Smith
The previous patch was accidentally not resetting the boolean limit flags to false for retries. Fixed in v2. -- Kind Regards, Peter Smith. Fujitsu Australia v2-0001-logicalrep_worker_launch-limit-checks.patch Description: Binary data

Re: Adding a LogicalRepWorker type field

2023-08-11 Thread Peter Smith
On Fri, Aug 11, 2023 at 9:13 PM Amit Kapila wrote: > > On Fri, Aug 11, 2023 at 3:41 PM Peter Smith wrote: > > > > On Fri, Aug 11, 2023 at 7:33 PM Amit Kapila wrote: > > > > > > On Thu, Aug 10, 2023 at 7:50 AM Peter Smith wrote: > > > > > > > > > > > > > > * If you do the above then there won't

Re: run pgindent on a regular basis / scripted manner

2023-08-11 Thread Andres Freund
Hi, On 2023-08-11 18:30:02 -0400, Tom Lane wrote: > Peter Geoghegan writes: > > On Fri, Aug 11, 2023 at 2:25 PM Andres Freund wrote: > >> We could a test that fails when there's some mis-indented code. That seems > >> like it might catch things earlier? > > +1 for including this in CI tests I

Re: run pgindent on a regular basis / scripted manner

2023-08-11 Thread Jelte Fennema
On Fri, 11 Aug 2023 at 23:00, Peter Geoghegan wrote: > I'm starting to have doubts about this policy. There have now been > quite a few follow-up "fixes" to indentation issues that koel > complained about. I think one thing that would help a lot in reducing the is for committers to set up the

Re: run pgindent on a regular basis / scripted manner

2023-08-11 Thread Tom Lane
Peter Geoghegan writes: > I'm starting to have doubts about this policy. There have now been > quite a few follow-up "fixes" to indentation issues that koel > complained about. None of these fixups have been included in > .git-blame-ignore-revs. If things continue like this then "git blame" > is

Re: run pgindent on a regular basis / scripted manner

2023-08-11 Thread Tom Lane
Peter Geoghegan writes: > My workflow up until now has avoiding making updates to typedefs.list > in patches. I only update typedefs locally, for long enough to indent > my code. The final patch doesn't retain any typedefs.list changes. Yeah, I've done the same and will have to stop. > I guess

Re: run pgindent on a regular basis / scripted manner

2023-08-11 Thread Peter Geoghegan
On Fri, Aug 11, 2023 at 3:30 PM Tom Lane wrote: > No. I presume koel is using src/tools/pgindent/typedefs.list, > which has always been the "canonical" list but up to now we've > been lazy about maintaining it. Part of the new regime is that > typedefs.list should now be updated on-the-fly by

Re: run pgindent on a regular basis / scripted manner

2023-08-11 Thread Tom Lane
Peter Geoghegan writes: > On Fri, Aug 11, 2023 at 2:25 PM Andres Freund wrote: >> We could a test that fails when there's some mis-indented code. That seems >> like it might catch things earlier? +1 for including this in CI tests > It definitely would. That would go a long way towards

Re: run pgindent on a regular basis / scripted manner

2023-08-11 Thread Peter Geoghegan
On Fri, Aug 11, 2023 at 2:25 PM Andres Freund wrote: > > I don't think that it makes sense to invent yet another rule for > > .git-blame-ignore-revs, though. Will we need another buildfarm member > > to enforce that rule, too? > > We could a test that fails when there's some mis-indented code.

Re: AssertLog instead of Assert in some places

2023-08-11 Thread Peter Geoghegan
On Fri, Aug 11, 2023 at 2:04 PM Andres Freund wrote: > Where supported, a crash (distinguishing from assertion failures) will now > report something like: > > process with pid: 2900527 received signal: SIGSEGV, si_code: 1, si_addr: > 0xdeadbeef > [0x5628ec45212f]

Re: run pgindent on a regular basis / scripted manner

2023-08-11 Thread Andres Freund
Hi, On 2023-08-11 13:59:40 -0700, Peter Geoghegan wrote: > On Sat, Jun 17, 2023 at 7:08 AM Andrew Dunstan wrote: > > I have set up a new buildfarm animal called koel which will run the module. > > I'm starting to have doubts about this policy. There have now been > quite a few follow-up "fixes"

Re: AssertLog instead of Assert in some places

2023-08-11 Thread Andres Freund
Hi, On 2023-08-11 13:19:34 -0700, Peter Geoghegan wrote: > On Fri, Aug 11, 2023 at 12:26 PM Andres Freund wrote: > > > For example, dealing with core dumps left behind by the regression > > > tests can be annoying. > > > > Hm. I don't have a significant problem with that. But I can see it being

Re: [PATCH] Support static linking against LLVM

2023-08-11 Thread Marcelo Juchem
In my case, my product has a very controlled environment. We build all our infrastructure from source and we avoid dynamic linking by design, except where technically not viable (e.g.: pgsql extensions). LLVM is one of the libraries we're specifically required to statically link. Unfortunately I

Re: [PATCH] Support static linking against LLVM

2023-08-11 Thread Marcelo Juchem
Andres, Tom, I found your names in the git history for JIT and LLVM. Any chance one of you could take a look at the patch? -mj On Thu, Aug 10, 2023 at 2:45 PM Marcelo Juchem wrote: > By default, PostgreSQL doesn't explicitly choose whether to link > statically or dynamically against LLVM when

Re: run pgindent on a regular basis / scripted manner

2023-08-11 Thread Peter Geoghegan
On Sat, Jun 17, 2023 at 7:08 AM Andrew Dunstan wrote: > I have set up a new buildfarm animal called koel which will run the module. I'm starting to have doubts about this policy. There have now been quite a few follow-up "fixes" to indentation issues that koel complained about. None of these

Re: walsender "wakeup storm" on PG16, likely because of bc971f4025c (Optimize walsender wake up logic using condition variables)

2023-08-11 Thread Andres Freund
Hi, On 2023-08-12 07:51:09 +1200, Thomas Munro wrote: > Oh, I see what's happening. Maybe commit b91dd9de wasn't the best > idea, but bc971f4025c broke an assumption, since it doesn't use > ConditionVariableSleep(). That is confusing the signal forwarding > logic which expects to find our entry

Re: AssertLog instead of Assert in some places

2023-08-11 Thread Peter Geoghegan
On Fri, Aug 11, 2023 at 12:26 PM Andres Freund wrote: > > For example, dealing with core dumps left behind by the regression > > tests can be annoying. > > Hm. I don't have a significant problem with that. But I can see it being > problematic. Unfortunately, short of preventing core dumps from

Re: walsender "wakeup storm" on PG16, likely because of bc971f4025c (Optimize walsender wake up logic using condition variables)

2023-08-11 Thread Thomas Munro
On Sat, Aug 12, 2023 at 5:51 AM Andres Freund wrote: > On 2023-08-11 15:31:43 +0200, Tomas Vondra wrote: > > It seems to me the issue is in WalSndWait, which was reworked to use > > ConditionVariableCancelSleep() in bc971f4025c. The walsenders end up > > waking each other in a busy loop, until

Re: [PATCH] psql: Add tab-complete for optional view parameters

2023-08-11 Thread David Zhang
Applied v3 patch to master and verified it with below commands, #Alter view postgres=# alter view v ALTER COLUMN  OWNER TO  RENAME    RESET (   SET postgres=# alter view v set (   SCHEMA postgres=# alter view v set ( CHECK_OPTION  SECURITY_BARRIER  SECURITY_INVOKER

Re: AssertLog instead of Assert in some places

2023-08-11 Thread Andres Freund
Hi, On 2023-08-11 11:56:27 -0700, Peter Geoghegan wrote: > On Fri, Aug 11, 2023 at 11:23 AM Andres Freund wrote: > > > Couldn't you say the same thing about defensive "can't happen" ERRORs? > > > They are essentially a form of assertion that isn't limited to > > > assert-enabled builds. > > > >

Re: AssertLog instead of Assert in some places

2023-08-11 Thread Peter Geoghegan
On Fri, Aug 11, 2023 at 11:23 AM Andres Freund wrote: > > Couldn't you say the same thing about defensive "can't happen" ERRORs? > > They are essentially a form of assertion that isn't limited to > > assert-enabled builds. > > Yes. A lot of them I hate them with the passion of a thousand suns ;).

Re: AssertLog instead of Assert in some places

2023-08-11 Thread Andres Freund
Hi, On 2023-08-11 11:14:34 -0700, Peter Geoghegan wrote: > On Fri, Aug 11, 2023 at 10:57 AM Andres Freund wrote: > > I am quite strongly against this. This will lead to assertions being hit in > > tests without that being noticed, e.g. because they happen in a background > > process that just

Re: AssertLog instead of Assert in some places

2023-08-11 Thread Peter Geoghegan
On Fri, Aug 11, 2023 at 10:57 AM Andres Freund wrote: > I am quite strongly against this. This will lead to assertions being hit in > tests without that being noticed, e.g. because they happen in a background > process that just restarts. Couldn't you say the same thing about defensive "can't

Re: [PATCH] Support static linking against LLVM

2023-08-11 Thread Andres Freund
Hi, On 2023-08-11 12:59:27 -0500, Marcelo Juchem wrote: > In my case, my product has a very controlled environment. > We build all our infrastructure from source and we avoid dynamic linking by > design, except where technically not viable (e.g.: pgsql extensions). > > LLVM is one of the

Re: LLVM 16 (opaque pointers)

2023-08-11 Thread Andres Freund
Hi, On 2023-05-21 15:01:41 +1200, Thomas Munro wrote: > *For anyone working with this type of IR generation code and > questioning their sanity, I can pass on some excellent advice I got > from Andres: build LLVM yourself with assertions enabled, as they > catch some classes of silly mistake that

Re: [PoC] pg_upgrade: allow to upgrade publisher node

2023-08-11 Thread Bruce Momjian
On Fri, Aug 11, 2023 at 10:46:31AM +0530, Amit Kapila wrote: > On Thu, Aug 10, 2023 at 7:07 PM Masahiko Sawada wrote: > > What I imagined is that we do this check before > > check_and_dump_old_cluster() while the server is 'off'. Reading the > > slot state file would be simple and I guess we

Re: [PoC] pg_upgrade: allow to upgrade publisher node

2023-08-11 Thread Bruce Momjian
On Fri, Aug 11, 2023 at 11:18:09AM +0530, Amit Kapila wrote: > On Fri, Aug 11, 2023 at 10:43 AM Julien Rouhaud wrote: > > I disagree. As I mentioned before any module registered in > > shared_preload_libraries can spawn background workers which can perform any > > activity. There were previous

Re: LLVM 16 (opaque pointers)

2023-08-11 Thread Andres Freund
Hi, On 2023-08-10 16:56:54 +0200, Ronan Dunklau wrote: > I tried my hand at backporting it to previous versions, and not knowing > anything about it made me indeed question my sanity. It's quite easy for PG > 15, 14, 13. PG 12 is nothing insurmontable either, but PG 11 is a bit hairier > most

Re: AssertLog instead of Assert in some places

2023-08-11 Thread Andres Freund
Hi, On 2023-08-11 17:59:37 +0530, Ashutosh Bapat wrote: > Most of the Asserts are recoverable by rolling back the transaction > without crashing the backend. So an elog(ERROR, ) is enough. But just > by themselves elogs are compiled into non-debug binary and the > condition check can waste CPU

Re: walsender "wakeup storm" on PG16, likely because of bc971f4025c (Optimize walsender wake up logic using condition variables)

2023-08-11 Thread Andres Freund
Hi, On 2023-08-11 15:31:43 +0200, Tomas Vondra wrote: > That's an awful lot of CPU for a cluster doing essentially "nothing" > (there's no WAL to decode/replicate, etc.). It usually disappears after > a couple seconds, but sometimes it's a rather persistent state. Ugh, that's not great. > The

Re: [PATCH] Support static linking against LLVM

2023-08-11 Thread Andres Freund
Hi, On 2023-08-10 14:45:47 -0500, Marcelo Juchem wrote: > By default, PostgreSQL doesn't explicitly choose whether to link > statically or dynamically against LLVM when LLVM JIT is enabled (e.g.: > `./configure --with-llvm`). > > `llvm-config` will choose to dynamically link by default. > > In

Re: A failure in 031_recovery_conflict.pl on Debian/s390x

2023-08-11 Thread Christoph Berg
Re: Thomas Munro > On Thu, Aug 10, 2023 at 9:15 PM Christoph Berg wrote: > > No XXX lines this time either, but I've seen then im logfiles that > > went through successfully. > > Do you still have the data directories around from that run, so we can > see if the expected Heap2/PRUNE was actually

Re: Schema variables - new implementation for Postgres 15

2023-08-11 Thread Dmitry Dolgov
> On Thu, Aug 03, 2023 at 08:15:13AM +0200, Pavel Stehule wrote: > Hi > > fresh rebase Thanks for continuing efforts. The new patch structure looks better to me (although the boundary between patches 0001 and 0002 is somewhat fuzzy, e.g. the function NameListToString is used already in the first

Re: generic plans and "initial" pruning

2023-08-11 Thread Amit Langote
On Fri, Aug 11, 2023 at 14:31 Amit Langote wrote: > On Wed, Aug 9, 2023 at 1:05 AM Robert Haas wrote: > > On Tue, Aug 8, 2023 at 10:32 AM Amit Langote > wrote: > > > But should ExecInitNode() subroutines return the partially initialized > > > PlanState node or NULL on detecting invalidation?

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

2023-08-11 Thread Melih Mutlu
Hi Peter, Peter Smith , 11 Ağu 2023 Cum, 01:26 tarihinde şunu yazdı: > No, I meant what I wrote there. When I ran the tests the HEAD included > the v25-0001 refactoring patch, but v26 did not yet exist. > > For now, we are only performance testing the first > "Reuse-Tablesyc-Workers" patch, but

walsender "wakeup storm" on PG16, likely because of bc971f4025c (Optimize walsender wake up logic using condition variables)

2023-08-11 Thread Tomas Vondra
Hi, while working on some logical replication stuff, I noticed that on PG16 I often end up with a completely idle publisher (no user activity), that however looks like this in top: ... %CPU COMMAND ... 17.9 postgres: walsender user test ::1(43064) START_REPLICATION ... 16.6

Re: Report planning memory in EXPLAIN ANALYZE

2023-08-11 Thread Ashutosh Bapat
On Fri, Aug 11, 2023 at 10:41 AM Andrey Lepikhov wrote: > >> > > > > This won't just affect planner but every subsystem of PostgreSQL, not > > just planner, unless we device a new context type for planner. > > > > My point is what's relevant here is how much net memory planner asked > > for.

Re: Inconsistent results with libc sorting on Windows

2023-08-11 Thread Noah Misch
On Fri, Aug 11, 2023 at 11:48:18AM +0200, Juan José Santamaría Flecha wrote: > On Fri, Aug 11, 2023 at 7:29 AM Noah Misch wrote: > > > > > The LCMapStringEx() solution is elegant. I do see > > > > https://learn.microsoft.com/en-us/windows/win32/intl/handling-sorting-in-your-applications > >

Re: Reducing memory consumed by RestrictInfo list translations in partitionwise join planning

2023-08-11 Thread Ashutosh Bapat
I spent some time on 4th point below but also looked at other points. Here's what I have found so far On Thu, Jul 27, 2023 at 7:35 PM Ashutosh Bapat wrote: > > 1. The patch uses RestrictInfo::required_relids as the key for > searching child RelOptInfos. I am not sure which of the two viz. >

AssertLog instead of Assert in some places

2023-08-11 Thread Ashutosh Bapat
Hi All, PostgreSQL code uses Assert() as a way to 1. document the assumption or conditions which must be true at a given place in code 2. make sure that some bug elsewhere does not break these assumptions or rules 3. conditions which can not be (easily) induced by user actions E.g. following

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

2023-08-11 Thread vignesh C
On Fri, 11 Aug 2023 at 16:26, vignesh C wrote: > > On Wed, 9 Aug 2023 at 09:51, vignesh C wrote: > > > > Hi Melih, > > > > Here is a patch to help in getting the execution at various phases > > like: a) replication slot creation time, b) Wal reading c) Number of > > WAL records read d)

Re: Let's make PostgreSQL multi-threaded

2023-08-11 Thread Merlin Moncure
On Thu, Jul 27, 2023 at 8:28 AM David Geier wrote: > Hi, > > On 6/7/23 23:37, Andres Freund wrote: > > I think we're starting to hit quite a few limits related to the process > model, > > particularly on bigger machines. The overhead of cross-process context > > switches is inherently higher

Re: libpq compression (part 2)

2023-08-11 Thread Andrey M. Borodin
> On 10 Aug 2023, at 19:47, Jonah H. Harris wrote: > > Pinging to see if anyone has continued to work on this behind-the-scenes or > whether this is the latest patch set there is. It's still on my TODO list, but I haven't done much review cycles yet. And the patch series already needs

Re: Adding a LogicalRepWorker type field

2023-08-11 Thread Amit Kapila
On Fri, Aug 11, 2023 at 3:41 PM Peter Smith wrote: > > On Fri, Aug 11, 2023 at 7:33 PM Amit Kapila wrote: > > > > On Thu, Aug 10, 2023 at 7:50 AM Peter Smith wrote: > > > > > > > > > > > * If you do the above then there won't be a need to change the > > > > variable name

Re: [PATCH] Reuse Workers and Replication Slots during Logical Replication

2023-08-11 Thread vignesh C
On Wed, 9 Aug 2023 at 09:51, vignesh C wrote: > > Hi Melih, > > Here is a patch to help in getting the execution at various phases > like: a) replication slot creation time, b) Wal reading c) Number of > WAL records read d) subscription relation state change etc > Couple of observation while we

Re: Adding a LogicalRepWorker type field

2023-08-11 Thread Peter Smith
On Fri, Aug 11, 2023 at 7:33 PM Amit Kapila wrote: > > On Thu, Aug 10, 2023 at 7:50 AM Peter Smith wrote: > > > > > > > > * If you do the above then there won't be a need to change the > > > variable name is_parallel_apply_worker in logicalrep_worker_launch. > > > > > > > Done. > > > > I don't

Re: Inconsistent results with libc sorting on Windows

2023-08-11 Thread Juan José Santamaría Flecha
On Fri, Aug 11, 2023 at 7:29 AM Noah Misch wrote: > > The LCMapStringEx() solution is elegant. I do see > > https://learn.microsoft.com/en-us/windows/win32/intl/handling-sorting-in-your-applications > says, "If an application calls the function to create a sort key for a > string > containing

Re: Adding a LogicalRepWorker type field

2023-08-11 Thread Amit Kapila
On Thu, Aug 10, 2023 at 7:50 AM Peter Smith wrote: > > > > > * If you do the above then there won't be a need to change the > > variable name is_parallel_apply_worker in logicalrep_worker_launch. > > > > Done. > I don't think the addition of two new macros isTablesyncWorker() and

logicalrep_worker_launch -- counting/checking the worker limits

2023-08-11 Thread Peter Smith
While reviewing other threads I have been looking more closely at the the logicalrep_worker_launch() function. IMO the logic of that function seems not quite right. Here are a few things I felt are strange: 1. The function knows exactly what type of worker it is launching, but still, it is

Re: proposal: psql: show current user in prompt

2023-08-11 Thread Pavel Stehule
čt 10. 8. 2023 v 16:31 odesílatel Jelte Fennema napsal: > On Thu, 10 Aug 2023 at 14:44, Pavel Stehule > wrote: > > čt 10. 8. 2023 v 14:05 odesílatel Jelte Fennema > napsal: > >> That it is not rolled-back > >> in a case like this? > >> > >> BEGIN; > >> \set PROMPT '%N' > >> ROLLBACK; > > > > >

Re: logical decoding and replication of sequences, take 2

2023-08-11 Thread Ashutosh Bapat
On Tue, Aug 1, 2023 at 8:46 PM Tomas Vondra wrote: > > Anyway, I think this is "just" a matter of efficiency, not correctness. > IMHO there are bigger questions regarding the "going back" behavior > after apply restart. sequence_decode() has the following code /* Skip the change if already