Re: Syncrep and improving latency due to WAL throttling

2023-01-30 Thread Bharath Rupireddy
On Sat, Jan 28, 2023 at 6:06 AM Tomas Vondra wrote: > > > > > That's not the sole goal, from my end: I'd like to avoid writing out + > > flushing the WAL in too small chunks. Imagine a few concurrent vacuums or > > COPYs or such - if we're unlucky they'd each end up exceeding their > > "private"

Re: lockup in parallel hash join on dikkop (freebsd 14.0-current)

2023-01-30 Thread Thomas Munro
On Mon, Jan 30, 2023 at 6:36 PM Andres Freund wrote: > On 2023-01-30 15:22:34 +1300, Thomas Munro wrote: > > On Mon, Jan 30, 2023 at 6:26 AM Thomas Munro wrote: > > > out-of-order hazard > > > > I've been trying to understand how that could happen, but my CPU-fu is > > weak. Let me try to write

Re: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-30 Thread Amit Kapila
On Mon, Jan 30, 2023 at 12:38 PM Kyotaro Horiguchi wrote: > > At Mon, 30 Jan 2023 11:56:33 +0530, Amit Kapila > wrote in > > > > > > The GUC is not stored in a catalog, but.. oh... it is multiplied by > > > 1000. > > > > Which part of the patch you are referring to here? Isn't the check in > > W

Re: Check lateral references within PHVs for memoize cache keys

2023-01-30 Thread Richard Guo
On Tue, Jan 24, 2023 at 10:07 AM David Rowley wrote: > I'm surprised to see that it's only Memoize that ever makes use of > lateral_vars. I'd need a bit more time to process your patch, but one > additional thought I had was that I wonder if the following code is > still needed in nodeMemoize.c >

Re: Deadlock between logrep apply worker and tablesync worker

2023-01-30 Thread vignesh C
On Mon, 30 Jan 2023 at 13:00, houzj.f...@fujitsu.com wrote: > > On Monday, January 30, 2023 2:32 PM Amit Kapila > wrote: > > > > On Mon, Jan 30, 2023 at 9:20 AM vignesh C wrote: > > > > > > On Sat, 28 Jan 2023 at 11:26, Amit Kapila wrote: > > > > > > > > One thing that looks a bit odd is that

Re: Lazy JIT IR code generation to increase JIT speed with partitions

2023-01-30 Thread David Geier
Hi Dmitry, On 1/27/23 16:18, Dmitry Dolgov wrote: As I've noted off-list, there was noticeable difference in the dumped bitcode, which I haven't noticed since we were talking mostly about differences between executions of the same query. Thanks for the clarification and also thanks for helping

Re: Todo: Teach planner to evaluate multiple windows in the optimal order

2023-01-30 Thread Ankit Kumar Pandey
> On 30/01/23 11:01, John Naylor wrote: > Since David referred to L3 size as the starting point of a possible configuration parameter, that's actually cache-conscious. Okay, makes sense. I am correcting error on my part. > I'm not close enough to this thread to guess at the right direction

RE: Logical replication timeout problem

2023-01-30 Thread wangw.f...@fujitsu.com
On Mon, Jan 30, 2023 at 14:55 PM Amit Kapila wrote: > On Mon, Jan 30, 2023 at 10:36 AM wangw.f...@fujitsu.com > wrote: > > > > On Mon, Jan 30, 2023 11:37 AM Shi, Yu/侍 雨 > wrote: > > > On Sun, Jan 29, 2023 3:41 PM wangw.f...@fujitsu.com > > > wrote: > > > > Yes, I think you are right. > > Fixed

Re: Making Vars outer-join aware

2023-01-30 Thread Richard Guo
On Tue, Jan 24, 2023 at 4:38 AM Tom Lane wrote: > Richard, are you planning to review this any more? I'm getting > a little antsy to get it committed. For such a large patch, > it's surprising it's had so few conflicts to date. Sorry for the delayed reply. I don't have any more review commen

RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-30 Thread Takamichi Osumi (Fujitsu)
On Saturday, January 28, 2023 1:28 PM I wrote: > Attached the updated patch v24. Hi, I've conducted the rebase affected by the commit(1e8b61735c) by renaming the GUC to logical_replication_mode accordingly, because it's utilized in the TAP test of this time-delayed LR feature. There is no other c

MacOS: xsltproc fails with "warning: failed to load external entity"

2023-01-30 Thread Aleksander Alekseev
Hi hackers, I'm having some difficulties building the documentation on MacOS. I'm using ./full-build.sh script from [1] repository. It worked just fine for many years but since recently it started to fail like this: ``` /usr/bin/xsltproc --path . --stringparam pg.version '16devel' /Users/eax/pro

Re: old_snapshot_threshold bottleneck on replica

2023-01-30 Thread Maxim Orlov
On Fri, 27 Jan 2023 at 18:18, Robert Haas wrote: > > Interesting, but it's still not entirely clear to me from reading the > comments why we should think that this is safe. > In overall, I think this is safe, because we do not change algorithm here. More specific, threshold_timestamp have only u

Re: Add SHELL_EXIT_CODE to psql

2023-01-30 Thread Maxim Orlov
Unfortunately, there is a fail in FreeBSD https://cirrus-ci.com/task/6466749487382528 Maybe, this patch is need to be rebased? -- Best regards, Maxim Orlov.

Re: Considering additional sort specialisation functions for PG16

2023-01-30 Thread John Naylor
I wrote: > On Thu, Jan 26, 2023 at 7:15 PM David Rowley wrote: > > I think the slower sorts I found in [2] could also be partially caused > > by the current sort specialisation comparators re-comparing the > > leading column during a tie-break. I've not gotten around to disabling > > the sort spe

Re: Deadlock between logrep apply worker and tablesync worker

2023-01-30 Thread vignesh C
On Mon, 30 Jan 2023 at 13:00, houzj.f...@fujitsu.com wrote: > > On Monday, January 30, 2023 2:32 PM Amit Kapila > wrote: > > > > On Mon, Jan 30, 2023 at 9:20 AM vignesh C wrote: > > > > > > On Sat, 28 Jan 2023 at 11:26, Amit Kapila wrote: > > > > > > > > One thing that looks a bit odd is that

RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-30 Thread Takamichi Osumi (Fujitsu)
On Monday, January 30, 2023 12:02 PM Kyotaro Horiguchi wrote: > At Sat, 28 Jan 2023 04:28:29 +, "Takamichi Osumi (Fujitsu)" > wrote in > > On Friday, January 27, 2023 8:00 PM Amit Kapila > wrote: > > > So, you have changed min_apply_delay from int64 to int32, but you > > > haven't mentioned

Re: Generating code for query jumbling through gen_node_support.pl

2023-01-30 Thread Michael Paquier
On Fri, Jan 27, 2023 at 11:59:47AM +0900, Michael Paquier wrote: > Using that, I can compile the following results for various cases (-O2 > and compute_query_id=on): > query | mode | iterations | avg_runtime_ns | > avg_jumble_ns > -++--

Re: [DOCS] Stats views and functions not in order?

2023-01-30 Thread Peter Eisentraut
On 30.01.23 07:12, Peter Smith wrote: Meanwhile, this pagination topic has strayed far away from the original $SUBJECT, so I guess since there is nothing else pending this thread's CF entry [1] can just be marked as "Committed" now? done

Re: Generating code for query jumbling through gen_node_support.pl

2023-01-30 Thread Peter Eisentraut
On 27.01.23 04:07, Michael Paquier wrote: On Thu, Jan 26, 2023 at 09:39:05AM +0100, Peter Eisentraut wrote: There are a couple of repetitive comments, like "typmod and collation information are irrelevant for the query jumbling". This applies to all nodes, so we don't need to repeat it for a nu

Re: Generating code for query jumbling through gen_node_support.pl

2023-01-30 Thread Peter Eisentraut
On 27.01.23 03:59, Michael Paquier wrote: At the end, that would be unnoticeable for the average user, I guess, but here are the numbers I get on my laptop :) Personally, I think we do not want the two jumble methods in parallel. Maybe there are other opinions. I'm going to set this thread as

Re: Allow logical replication to copy tables in binary format

2023-01-30 Thread Melih Mutlu
Hi Bharath, Thanks for reviewing. Bharath Rupireddy , 18 Oca 2023 Çar, 10:17 tarihinde şunu yazdı: > On Thu, Jan 12, 2023 at 1:53 PM Melih Mutlu > wrote: > 1. The performance numbers posted upthread [1] look impressive for the > use-case tried, that's a 2.25X improvement or 55.6% reduction in >

Re: MacOS: xsltproc fails with "warning: failed to load external entity"

2023-01-30 Thread Aleksander Alekseev
Hi hackers, > At this point I could use a friendly piece of advice from the community. I've found a solution: ``` export SGML_CATALOG_FILES=/usr/local/etc/xml/catalog export XMLLINT="xmllint --catalogs" export XSLTPROC="xsltproc --catalogs" ``` I will submit a patch for the documentation in a b

Re: Assertion failure in SnapBuildInitialSnapshot()

2023-01-30 Thread Amit Kapila
On Thu, Dec 8, 2022 at 8:17 AM Masahiko Sawada wrote: > > On Mon, Nov 21, 2022 at 4:31 PM Amit Kapila wrote: > > One idea to fix this issue is that in > ReplicationSlotsComputeRequiredXmin(), we compute the minimum xmin > while holding both ProcArrayLock and ReplicationSlotControlLock, and > rele

Re: Assertion failure in SnapBuildInitialSnapshot()

2023-01-30 Thread Amit Kapila
On Mon, Jan 30, 2023 at 11:34 AM Amit Kapila wrote: > > I have reproduced it manually. For this, I had to manually make the > debugger call ReplicationSlotsComputeRequiredXmin(false) via path > SnapBuildProcessRunningXacts()->LogicalIncreaseXminForSlot()->LogicalConfirmReceivedLocation() > ->Repli

Re: Assertion failure in SnapBuildInitialSnapshot()

2023-01-30 Thread Amit Kapila
On Fri, Jan 27, 2023 at 4:31 PM Hayato Kuroda (Fujitsu) wrote: > > Thank you for making the patch! I'm still considering whether this approach is > correct, but I can put a comment to your patch anyway. > > ``` > - Assert(!already_locked || LWLockHeldByMe(ProcArrayLock)); > - > - if (!

Re: Fix GUC_NO_SHOW_ALL test scenario in 003_check_guc.pl

2023-01-30 Thread Nitin Jadhav
> I kind of think this is a lot of unnecessary work. The case that is > problematic is a GUC that's marked GUC_NO_SHOW_ALL but not marked > GUC_NOT_IN_SAMPLE. There aren't any of those, and I don't think there > are likely to be any in future, because it doesn't make a lot of sense. > Why don't w

Re: MacOS: xsltproc fails with "warning: failed to load external entity"

2023-01-30 Thread Aleksander Alekseev
Hi hackers, > I've found a solution: > > ``` > export SGML_CATALOG_FILES=/usr/local/etc/xml/catalog > export XMLLINT="xmllint --catalogs" > export XSLTPROC="xsltproc --catalogs" > ``` > > I will submit a patch for the documentation in a bit, after I'll check > it properly. PFA the patch. I don't

Re: Deadlock between logrep apply worker and tablesync worker

2023-01-30 Thread vignesh C
On Mon, 30 Jan 2023 at 13:00, houzj.f...@fujitsu.com wrote: > > On Monday, January 30, 2023 2:32 PM Amit Kapila > wrote: > > > > On Mon, Jan 30, 2023 at 9:20 AM vignesh C wrote: > > > > > > On Sat, 28 Jan 2023 at 11:26, Amit Kapila wrote: > > > > > > > > One thing that looks a bit odd is that

Re: generate_series for timestamptz and time zone problem

2023-01-30 Thread Przemysław Sztoch
Gurjeet Singh wrote on 30.01.2023 08:18: On Sat, Nov 12, 2022 at 10:44 AM Tom Lane wrote: However, what it shouldn't be is part of this patch. It's worth pushing it separately to have a record of that decision. I've now done that, so you'll need to rebase to remove that delta. I looked over

Re: Allow logical replication to copy tables in binary format

2023-01-30 Thread Bharath Rupireddy
On Mon, Jan 30, 2023 at 4:19 PM Melih Mutlu wrote: > Thanks for providing an updated patch. >> On Thu, Jan 12, 2023 at 1:53 PM Melih Mutlu wrote: >> 1. The performance numbers posted upthread [1] look impressive for the >> use-case tried, that's a 2.25X improvement or 55.6% reduction in >> exec

Re: Assertion failure in SnapBuildInitialSnapshot()

2023-01-30 Thread Masahiko Sawada
On Mon, Jan 30, 2023 at 8:30 PM Amit Kapila wrote: > > On Fri, Jan 27, 2023 at 4:31 PM Hayato Kuroda (Fujitsu) > wrote: > > > > Thank you for making the patch! I'm still considering whether this approach > > is > > correct, but I can put a comment to your patch anyway. > > > > ``` > > - As

RE: Perform streaming logical transactions by background workers and parallel apply

2023-01-30 Thread Hayato Kuroda (Fujitsu)
Dear Hou, Thank you for updating the patch! I checked your replies and new patch, and it seems good. Currently I have no comments Best Regards, Hayato Kuroda FUJITSU LIMITED

Re: [PATCH] Use indexes on the subscriber when REPLICA IDENTITY is full on the publisher

2023-01-30 Thread Amit Kapila
On Fri, Jan 27, 2023 at 6:32 PM Önder Kalacı wrote: >> >> I suppose the options are: >> 1. use regular planner uniformly >> 2. use regular planner only when there's no replica identity (or >> configurable?) >> 3. only use low-level functions >> 4. keep using sequential scans for every single upda

Re: JSONPath Child Operator?

2023-01-30 Thread Filipp Krylov
Hi David, On 2022-11-10 21:55, David E. Wheeler wrote: My question: Are there plans to support square bracket syntax for JSON object field name strings like this? Or to update to follow the standard as it’s finalized? This syntax is a part of "jsonpath syntax extensions" patchset: https://ww

Re: dynamic result sets support in extended query protocol

2023-01-30 Thread Alvaro Herrera
On 2022-Nov-22, Peter Eisentraut wrote: > I added tests using the new psql \bind command to test this functionality in > the extended query protocol, which showed that this got broken since I first > wrote this patch. This "blame" is on the pipeline mode in libpq patch > (acb7e4eb6b1c614c68a62fb3

Re: HOT chain validation in verify_heapam()

2023-01-30 Thread Himanshu Upadhyaya
Hi Hackers, On Sun, Jan 22, 2023 at 8:48 PM Himanshu Upadhyaya < upadhyaya.himan...@gmail.com> wrote: > > The test if (pred_in_progress || TransactionIdDidCommit(curr_xmin)) >> seems wrong to me. Shouldn't it be &&? Has this code been tested at >> all? It doesn't seem to have a test case. Some o

RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-30 Thread Takamichi Osumi (Fujitsu)
On Monday, January 30, 2023 7:05 PM I wrote: > On Saturday, January 28, 2023 1:28 PM I wrote: > > Attached the updated patch v24. > I've conducted the rebase affected by the commit(1e8b61735c) by renaming > the GUC to logical_replication_mode accordingly, because it's utilized in the > TAP test of

Re: Amcheck verification of GiST and GIN

2023-01-30 Thread Aleksander Alekseev
Hi Andrey, > Thanks! I also found out that there was a CI complaint about amcheck.h > not including some necessary stuff. Here's a version with a fix for > that. Thanks for the updated patchset. One little nitpick I have is that the tests cover only cases when all the checks pass successfully. T

Re: Perform streaming logical transactions by background workers and parallel apply

2023-01-30 Thread Masahiko Sawada
On Mon, Jan 30, 2023 at 3:23 PM houzj.f...@fujitsu.com wrote: > > On Monday, January 30, 2023 12:13 PM Peter Smith > wrote: > > > > Here are my review comments for v88-0002. > > Thanks for your comments. > > > > > == > > General > > > > 1. > > The test cases are checking the log content but

Re: [PoC] Improve dead tuple storage for lazy vacuum

2023-01-30 Thread Masahiko Sawada
On Mon, Jan 30, 2023 at 1:31 PM John Naylor wrote: > > On Sun, Jan 29, 2023 at 9:50 PM Masahiko Sawada wrote: > > > > On Sat, Jan 28, 2023 at 8:33 PM John Naylor > > wrote: > > > > The first implementation should be simple, easy to test/verify, easy to > > > understand, and easy to replace. As

Re: cutting down the TODO list thread

2023-01-30 Thread Bruce Momjian
On Mon, Jan 30, 2023 at 01:13:45PM +0700, John Naylor wrote: > On Tue, Jan 24, 2023 at 11:57 PM Bruce Momjian wrote: > > I think we just point them at the TODO list and they will read the top > > of the list first, no?  I think you are right that we updated the top of > > the TODO but didn't updat

Re: Non-superuser subscription owners

2023-01-30 Thread Robert Haas
On Fri, Jan 27, 2023 at 5:56 PM Mark Dilger wrote: > If the owner cannot modify the subscription, then the owner degenerates into > a mere "run-as" user. Note that this isn't how things work now, and even if > we disallowed owners from modifying the connection string, there would still > be ot

Re: JSONPath Child Operator?

2023-01-30 Thread David E . Wheeler
On Jan 30, 2023, at 08:17, Filipp Krylov wrote: >> My question: Are there plans to support square bracket syntax for JSON >> object field name strings like this? Or to update to follow the standard as >> it’s finalized? > > This syntax is a part of "jsonpath syntax extensions" patchset: > htt

Re: Timeline ID hexadecimal format

2023-01-30 Thread Sébastien Lardière
On 27/01/2023 15:55, Peter Eisentraut wrote: On 27.01.23 14:52, Sébastien Lardière wrote: The attached patch proposes to change the format of timelineid from %u to %X. I think your complaint has merit.  But note that if we did a change like this, then log files or reports from different versi

Re: Non-superuser subscription owners

2023-01-30 Thread Mark Dilger
> On Jan 30, 2023, at 7:44 AM, Robert Haas wrote: > > And if we suppose that > that already works and is safe, well then what's the case where I do > need a run-as user? A) Alice publishes tables, and occasionally adds new tables to existing publications. B) Bob manages subscriptions, and p

Re: MacOS: xsltproc fails with "warning: failed to load external entity"

2023-01-30 Thread Tom Lane
Aleksander Alekseev writes: >> I've found a solution: >> >> ``` >> export SGML_CATALOG_FILES=/usr/local/etc/xml/catalog >> export XMLLINT="xmllint --catalogs" >> export XSLTPROC="xsltproc --catalogs" >> ``` Hmm, there is no such directory on my Mac, and indeed this recipe does not work here. I

Re: Making Vars outer-join aware

2023-01-30 Thread Tom Lane
Richard Guo writes: > Sorry for the delayed reply. I don't have any more review comments at > the moment, except a nitpicking one. > In optimizer/README at line 729 there is a query as > SELECT * FROM a > LEFT JOIN (SELECT * FROM b WHERE b.z = 1) ss ON (a.x = b.y) > WHERE a.x = 1;

Re: pub/sub - specifying optional parameters without values.

2023-01-30 Thread Tom Lane
Peter Smith writes: > The v3 patch LGTM (just for the logical replication commands). Pushed then. regards, tom lane

Re: Deadlock between logrep apply worker and tablesync worker

2023-01-30 Thread vignesh C
On Mon, 30 Jan 2023 at 17:30, vignesh C wrote: > > On Mon, 30 Jan 2023 at 13:00, houzj.f...@fujitsu.com > wrote: > > > > On Monday, January 30, 2023 2:32 PM Amit Kapila > > wrote: > > > > > > On Mon, Jan 30, 2023 at 9:20 AM vignesh C wrote: > > > > > > > > On Sat, 28 Jan 2023 at 11:26, Amit Ka

Re: Optimizing PostgreSQL with LLVM's PGO+LTO

2023-01-30 Thread João Paulo Labegalini de Carvalho
> What compiler / version / flags / OS did you try? > I am running experiment on a machine with: - Intel(R) Xeon(R) Platinum 8268 CPU @ 2.90GHz - Ubuntu 18.04.6 LTS - LLVM/Clang 15.0.6 (build from source) These are the flags I am using: CFLAGS = -O3 -fuse-ld=lld -gline-tables-only -fpr

Re: Non-superuser subscription owners

2023-01-30 Thread Robert Haas
On Mon, Jan 30, 2023 at 11:11 AM Mark Dilger wrote: > > On Jan 30, 2023, at 7:44 AM, Robert Haas wrote: > > > > And if we suppose that > > that already works and is safe, well then what's the case where I do > > need a run-as user? > > A) Alice publishes tables, and occasionally adds new tables t

Re: meson: Optionally disable installation of test modules

2023-01-30 Thread Andres Freund
Hi, On 2023-01-30 08:37:42 +0100, Peter Eisentraut wrote: > One open issue (IMO) with the meson build system is that it installs the > test modules under src/test/modules/ as part of a normal installation. This > is because there is no way to set up up the build system to install extra > things on

Re: Optimizing PostgreSQL with LLVM's PGO+LTO

2023-01-30 Thread Andres Freund
Hi, On 2023-01-30 10:24:02 -0700, João Paulo Labegalini de Carvalho wrote: > > What compiler / version / flags / OS did you try? > > > > I am running experiment on a machine with: > >- Intel(R) Xeon(R) Platinum 8268 CPU @ 2.90GHz >- Ubuntu 18.04.6 LTS >- LLVM/Clang 15.0.6 (build from

Re: Add n_tup_newpage_upd to pg_stat table views

2023-01-30 Thread Corey Huinker
On Fri, Jan 27, 2023 at 6:55 PM Andres Freund wrote: > Hi, > > On 2023-01-27 18:23:39 -0500, Corey Huinker wrote: > > This patch adds the n_tup_newpage_upd to all the table stat views. > > > > Just as we currently track HOT updates, it should be beneficial to track > > updates where the new tuple

Re: Add n_tup_newpage_upd to pg_stat table views

2023-01-30 Thread Andres Freund
Hi, On 2023-01-30 13:40:15 -0500, Corey Huinker wrote: > I must be missing something, I only see the check for running out of space, > not the check for exhausting line pointers. I agree dividing them would be > interesting. See PageGetHeapFreeSpace(), particularly the header comment and the MaxH

Re: Non-superuser subscription owners

2023-01-30 Thread Mark Dilger
> On Jan 30, 2023, at 9:26 AM, Robert Haas wrote: > > First, it doesn't seem to make a lot of sense to have one person > managing the publications and someone else managing the subscriptions, > and especially if those parties are mutually untrusting. I can't think > of any real reason to set t

Re: Introduce "log_connection_stages" setting.

2023-01-30 Thread Sergey Dudoladov
Hello, hackers Thank you for the reviews. I've modified the patch to incorporate your suggestions: + flag bits are now used to encode different connection stages + failing tests are now fixed. It was not a keyword issue but rather "check_log_connection_messages" not allocating memory properly i

Re: MacOS: xsltproc fails with "warning: failed to load external entity"

2023-01-30 Thread Aleksander Alekseev
Hi Tom, Thanks for the feedback. > Hmm, there is no such directory on my Mac, and indeed this recipe > does not work here. I tried to transpose it to MacPorts by > substituting /opt/local/etc/xml/catalog, which does exist --- but > the recipe still doesn't work. Well, that's a bummer. > What w

Re: Non-superuser subscription owners

2023-01-30 Thread Mark Dilger
> On Jan 30, 2023, at 9:26 AM, Robert Haas wrote: > > So basically this doesn't really feel like a valid scenario to me. > We're supposed to believe that Alice is hostile to Bob, but the > superuser doesn't seem to have thought very carefully about how Bob is > supposed to defend himself again

Re: postgres_fdw uninterruptible during connection establishment / ProcSignalBarrier

2023-01-30 Thread Nathan Bossart
On Mon, Jan 23, 2023 at 07:28:06PM -0800, Andres Freund wrote: > After a tiny bit further polishing, and after separately pushing a resource > leak fix for walrcv_connect(), I pushed this. My colleague Robins Tharakan (CC'd) noticed crashes when testing recent commits, and he traced it back to e46

Re: Non-superuser subscription owners

2023-01-30 Thread Robert Haas
On Mon, Jan 30, 2023 at 1:46 PM Mark Dilger wrote: > I have a grim view of the requirement that publishers and subscribers trust > each other. Even when they do trust each other, they can firewall attacks by > acting as if they do not. I think it's OK if the CREATE PUBLICATION user doesn't par

Re: recovery modules

2023-01-30 Thread Nathan Bossart
On Mon, Jan 30, 2023 at 04:51:38PM +0900, Michael Paquier wrote: > Now, I find this part, where we use a double pointer to allow the > module initialization to create and give back a private area, rather > confusing, and I think that this could be bug-prone, as well. Once > you incorporate some da

Making background psql nicer to use in tap tests

2023-01-30 Thread Andres Freund
Hi, Plenty tap tests require a background psql. But they're pretty annoying to use. I think the biggest improvement would be an easy way to run a single query and get the result of that query. Manually having to pump_until() is awkward and often leads to hangs/timeouts, instead of test failures,

Re: recovery modules

2023-01-30 Thread Andres Freund
Hi, On 2023-01-30 16:51:38 +0900, Michael Paquier wrote: > On Fri, Jan 27, 2023 at 10:27:29PM -0800, Nathan Bossart wrote: > > Here is a work-in-progress patch set for adjusting the archive modules > > interface. Is this roughly what you had in mind? > > I have been catching up with what is happ

Re: postgres_fdw uninterruptible during connection establishment / ProcSignalBarrier

2023-01-30 Thread Andres Freund
Hi, On 2023-01-30 11:30:08 -0800, Nathan Bossart wrote: > On Mon, Jan 23, 2023 at 07:28:06PM -0800, Andres Freund wrote: > > After a tiny bit further polishing, and after separately pushing a resource > > leak fix for walrcv_connect(), I pushed this. > > My colleague Robins Tharakan (CC'd) notice

Re: postgres_fdw uninterruptible during connection establishment / ProcSignalBarrier

2023-01-30 Thread Nathan Bossart
On Mon, Jan 30, 2023 at 11:49:37AM -0800, Andres Freund wrote: > Why don't the dblink tests catch this? Any chance you or Robins could prepare > a patch with fix and test, given that you know how to trigger this? It's trivially reproducible by calling 1-argument dblink_connect() multiple times an

Re: recovery modules

2023-01-30 Thread Nathan Bossart
On Mon, Jan 30, 2023 at 11:48:10AM -0800, Andres Freund wrote: > I don't think _PG_archive_module_init() should actually allocate a memory > context and do other similar initializations. Instead it should just return > 'const ArchiveModuleCallbacks*', typically a single line. > > Allocations etc s

Re: Making background psql nicer to use in tap tests

2023-01-30 Thread Tom Lane
Andres Freund writes: > It's annoyingly hard to wait for the result of a query in a generic way with > background_psql(), and more generally for psql. background_psql() uses -XAtq, > which means that we'll not get "status" output (like "BEGIN" or "(1 row)"), > and that queries not returning anythi

Re: BUG: Postgres 14 + vacuum_defer_cleanup_age + FOR UPDATE + UPDATE

2023-01-30 Thread Andres Freund
Hi, On 2023-01-10 21:32:54 +0100, Matthias van de Meent wrote: > On Tue, 10 Jan 2023 at 20:14, Andres Freund wrote: > > On 2023-01-10 15:03:42 +0100, Matthias van de Meent wrote: > > What precisely do you mean with "skew" here? Do you just mean that it'd > > take a > > long time until vacuum_def

Re: Add SHELL_EXIT_CODE to psql

2023-01-30 Thread Corey Huinker
> > > Unfortunately, there is a fail in FreeBSD > https://cirrus-ci.com/task/6466749487382528 > > Maybe, this patch is need to be rebased? > > That failure is in postgres_fdw, which this code doesn't touch. I'm not able to get to /tmp/cirrus-ci-build/build/testrun/postgres_fdw-running/regress/reg

Allow an extention to be updated without a script

2023-01-30 Thread Yugo NAGATA
Hi hackers, I propose to add a new option "updates_without_script" to extension's control file which a list of updates that do not need update script. This enables to update an extension by ALTER EXTENSION even if the extension module doesn't provide the update script. Currently, even when we do

Re: Non-superuser subscription owners

2023-01-30 Thread Mark Dilger
> On Jan 30, 2023, at 11:30 AM, Robert Haas wrote: > > CREATE SUBSCRIPTION > provides no tools at all for filtering the data that the subscriber > chooses to send. > > Now that can be changed, I suppose, and a run-as user would be one way > to make progress in that direction. But I'm not sure

Re: Non-superuser subscription owners

2023-01-30 Thread Robert Haas
On Fri, Jan 27, 2023 at 5:00 PM Andres Freund wrote: > > Or, another thought, maybe this should be checking for CREATE on the > > current database + also pg_create_subscription. That seems like it > > might be the right idea, actually. > > Yes, that seems like a good idea. Done in this version. I

Re: run pgindent on a regular basis / scripted manner

2023-01-30 Thread Bruce Momjian
On Sat, Jan 28, 2023 at 05:06:03PM -0800, Noah Misch wrote: > On Tue, Jan 24, 2023 at 02:04:02PM -0500, Bruce Momjian wrote: > > On Tue, Jan 24, 2023 at 09:54:57AM -0500, Tom Lane wrote: > > > As another example, the mechanisms we use to create the typedefs list > > > in the first place are pretty

Re: heapgettup() with NoMovementScanDirection unused in core?

2023-01-30 Thread Melanie Plageman
v2 attached On Fri, Jan 27, 2023 at 6:28 PM David Rowley wrote: > > On Sat, 28 Jan 2023 at 12:15, Tom Lane wrote: > > /* > > * Determine the net effect of two direction specifications. > > * This relies on having ForwardScanDirection = +1, BackwardScanDirection = > > -1, > > * and will proba

Re: Authentication fails for md5 connections if ~/.postgresql/postgresql.{crt and key} exist

2023-01-30 Thread Jacob Champion
On Fri, Jan 27, 2023 at 12:13 PM Cary Huang wrote: > > (Eventually I'd like to teach the server not to ask for a client > > certificate if it's not going to use it.) > > If clientcert is not requested by the server, but yet the client still > sends the certificate, the server will still verify it

Re: MacOS: xsltproc fails with "warning: failed to load external entity"

2023-01-30 Thread Tom Lane
Aleksander Alekseev writes: >> What we do actually have already is a recommendation to install >> appropriate MacPorts or Homebrew packages: >> https://www.postgresql.org/docs/devel/docguide-toolsets.html#DOCGUIDE-TOOLSETS-INST-MACOS >> and it works okay for me as long as I use MacPorts' version o

Re: Authentication fails for md5 connections if ~/.postgresql/postgresql.{crt and key} exist

2023-01-30 Thread Jacob Champion
On Sun, Jan 29, 2023 at 5:02 AM Jim Jones wrote: > On 27.01.23 21:13, Cary Huang wrote: > > But, if the server does request clientcert but client uses > "sslcertmode=disable" to connect and not give a certificate, it would > also result in authentication failure. In this case, we actually would >

Re: Allow an extention to be updated without a script

2023-01-30 Thread Tom Lane
Yugo NAGATA writes: > Currently, even when we don't need to execute any command to update an > extension from one version to the next, we need to provide an update > script that doesn't contain any command. Preparing such meaningless > files are sometimes annoying. If you have no update script, w

Re: postgres_fdw, dblink, and CREATE SUBSCRIPTION security

2023-01-30 Thread Jacob Champion
On Fri, Jan 27, 2023 at 1:08 PM Robert Haas wrote: > > 1) Forwarding the original ambient context along with the request, so > > the server can check it too. > > Right, so a protocol extension. Reasonable idea, but a big lift. Not > only do you need everyone to be running a new enough version of >

Re: Non-superuser subscription owners

2023-01-30 Thread Robert Haas
On Mon, Jan 30, 2023 at 3:27 PM Mark Dilger wrote: > That was an aspirational example in which there's infinite daylight between > the publisher and subscriber. I, too, doubt that's ever going to be > possible. But I still think we should aspire to some extra daylight between > the two. Perh

Re: postgres_fdw, dblink, and CREATE SUBSCRIPTION security

2023-01-30 Thread Robert Haas
On Mon, Jan 30, 2023 at 4:12 PM Jacob Champion wrote: > For our case, assuming that connections have side effects, one > solution could be for the client to signal to the server that the > connection should use in-band authentication only; i.e. fail the > connection if the credentials provided are

Re: Transparent column encryption

2023-01-30 Thread Jacob Champion
On Wed, Jan 25, 2023 at 11:00 AM Peter Eisentraut wrote: > > When writing the paragraph on RSA-OAEP I was reminded that we didn't > > really dig into the asymmetric/symmetric discussion. Assuming that most > > first-time users will pick the builtin CMK encryption method, do we > > still want to ha

Re: pub/sub - specifying optional parameters without values.

2023-01-30 Thread Peter Smith
On Tue, Jan 31, 2023 at 4:00 AM Tom Lane wrote: > > Peter Smith writes: > > The v3 patch LGTM (just for the logical replication commands). > > Pushed then. > Thanks for pushing the v3 patch. I'd forgotten about the 'streaming' option -- AFAIK this was previously a boolean parameter and so its [

Re: pub/sub - specifying optional parameters without values.

2023-01-30 Thread Tom Lane
Peter Smith writes: > I'd forgotten about the 'streaming' option -- AFAIK this was > previously a boolean parameter and so its [= value] part can also be > omitted. However, in PG16 streaming became an enum type > (on/off/parallel), and the value can still be omitted but that is not > really being

Re: recovery modules

2023-01-30 Thread Michael Paquier
On Mon, Jan 30, 2023 at 12:04:22PM -0800, Nathan Bossart wrote: > On Mon, Jan 30, 2023 at 11:48:10AM -0800, Andres Freund wrote: >> I don't think _PG_archive_module_init() should actually allocate a memory >> context and do other similar initializations. Instead it should just return >> 'const Arch

Re: heapgettup refactoring

2023-01-30 Thread Melanie Plageman
v7 attached On Fri, Jan 27, 2023 at 10:34 PM David Rowley wrote: > > "On Wed, 25 Jan 2023 at 10:17, Melanie Plageman > wrote: > > I've added a comment but I didn't include the function name in it -- I > > find it repetitive when the comments above functions do that -- however, > > I'm not strong

monitoring usage count distribution

2023-01-30 Thread Nathan Bossart
My colleague Jeremy Schneider (CC'd) was recently looking into usage count distributions for various workloads, and he mentioned that it would be nice to have an easy way to do $SUBJECT. I've attached a patch that adds a pg_buffercache_usage_counts() function. This function returns a row per poss

Re: Making background psql nicer to use in tap tests

2023-01-30 Thread Andres Freund
Hi, On 2023-01-30 15:06:46 -0500, Tom Lane wrote: > Andres Freund writes: > > It's annoyingly hard to wait for the result of a query in a generic way with > > background_psql(), and more generally for psql. background_psql() uses > > -XAtq, > > which means that we'll not get "status" output (lik

Re: generate_series for timestamptz and time zone problem

2023-01-30 Thread Tom Lane
Gurjeet Singh writes: > [ generate_series_with_timezone.v6.patch ] The cfbot isn't terribly happy with this. It looks like UBSan is detecting some undefined behavior. Possibly an uninitialized variable? regards, tom lane

Re: Perform streaming logical transactions by background workers and parallel apply

2023-01-30 Thread Peter Smith
On Mon, Jan 30, 2023 at 5:23 PM houzj.f...@fujitsu.com wrote: > > On Monday, January 30, 2023 12:13 PM Peter Smith > wrote: > > > > Here are my review comments for v88-0002. > > Thanks for your comments. > > > > > == > > General > > > > 1. > > The test cases are checking the log content but

Re: Introduce "log_connection_stages" setting.

2023-01-30 Thread Justin Pryzby
Thanks for updating the patch. It's currently failing check-world, due to a test that was added on January 23 (a9dc7f941): http://cfbot.cputube.org/sergey-dudoladov.html [19:15:57.101] Summary of Failures: [19:15:57.101] [19:15:57.101] 250/251 postgresql:ldap / ldap/002_bindpasswd ERROR

Re: postgres_fdw uninterruptible during connection establishment / ProcSignalBarrier

2023-01-30 Thread Robins Tharakan
Hi Andres, On Tue, 31 Jan 2023 at 06:31, Nathan Bossart wrote: > > On Mon, Jan 30, 2023 at 11:49:37AM -0800, Andres Freund wrote: > > Why don't the dblink tests catch this? Any chance you or Robins could > > prepare > > a patch with fix and test, given that you know how to trigger this? > > It'

Re: pg_upgrade test failure

2023-01-30 Thread Thomas Munro
On Thu, Jan 5, 2023 at 4:11 PM Thomas Munro wrote: > On Wed, Dec 7, 2022 at 7:15 AM Andres Freund wrote: > > On 2022-11-08 01:16:09 +1300, Thomas Munro wrote: > > > So [1] on its own didn't fix this. My next guess is that the attached > > > might help. > > > What is our plan here? This afaict is

Re: odd buildfarm failure - "pg_ctl: control file appears to be corrupt"

2023-01-30 Thread Anton A. Melnikov
Hello! On 24.11.2022 04:02, Thomas Munro wrote: On Thu, Nov 24, 2022 at 11:05 AM Tom Lane wrote: Thomas Munro writes: ERROR: calculated CRC checksum does not match value stored in file The attached draft patch fixes it. Tried to catch this error on my PC, but failed to do it within a re

Re: Assertion failure in SnapBuildInitialSnapshot()

2023-01-30 Thread Masahiko Sawada
On Mon, Jan 30, 2023 at 8:24 PM Amit Kapila wrote: > > On Thu, Dec 8, 2022 at 8:17 AM Masahiko Sawada wrote: > > > > On Mon, Nov 21, 2022 at 4:31 PM Amit Kapila wrote: > > > > One idea to fix this issue is that in > > ReplicationSlotsComputeRequiredXmin(), we compute the minimum xmin > > while h

Scan buffercache for a table

2023-01-30 Thread Amin
Hi, I am looking for function calls to scan the buffer cache for a table and find the cached pages. I want to find out which pages are cached and which of them are dirty. Having the relation id, how can I do that? I have gone through bufmgr.c and relcache.c, but could not find a way to get relatio

Re: Allow an extention to be updated without a script

2023-01-30 Thread Yugo NAGATA
Hi, Thank you for your comment. On Mon, 30 Jan 2023 16:05:52 -0500 Tom Lane wrote: > Yugo NAGATA writes: > > Currently, even when we don't need to execute any command to update an > > extension from one version to the next, we need to provide an update > > script that doesn't contain any comma

Re: Scan buffercache for a table

2023-01-30 Thread Justin Pryzby
On Mon, Jan 30, 2023 at 06:01:08PM -0800, Amin wrote: > Hi, > > I am looking for function calls to scan the buffer cache for a table and > find the cached pages. I want to find out which pages are cached and which > of them are dirty. Having the relation id, how can I do that? I have gone > throug

RE: Perform streaming logical transactions by background workers and parallel apply

2023-01-30 Thread houzj.f...@fujitsu.com
On Monday, January 30, 2023 10:20 PM Masahiko Sawada wrote: > > > I have one comment on v89 patch: > > + /* > +* Using 'immediate' mode returns false to cause a switch to > +* PARTIAL_SERIALIZE mode so that the remaining changes will > be serialized. > +*/ > +

  1   2   >