Decouple last important WAL record LSN from WAL insert locks

2022-11-23 Thread Bharath Rupireddy
Hi, While working on something else, I noticed that each WAL insert lock tracks its own last important WAL record's LSN (lastImportantAt) and both the bgwriter and checkpointer later computes the max value/server-wide last important WAL record's LSN via GetLastImportantRecPtr(). While doing so,

Re: Perform streaming logical transactions by background workers and parallel apply

2022-11-23 Thread Amit Kapila
On Tue, Nov 22, 2022 at 7:30 AM houzj.f...@fujitsu.com wrote: > Few minor comments and questions: 1. +static void +LogicalParallelApplyLoop(shm_mq_handle *mqh) { + for (;;) + { + void*data; + Size len; + + ProcessParallelApplyInterrupts(); ... ... + if (rc &

Re: Hash index build performance tweak from sorting

2022-11-23 Thread Simon Riggs
On Wed, 23 Nov 2022 at 13:04, David Rowley wrote: > After getting rid of the HashInsertState code and just adding bool > sorted to _hash_doinsert() and _hash_pgaddtup(), the resulting patch > is much more simple: Seems good to me and I wouldn't argue with any of your comments. > and v4

Re: Prefetch the next tuple's memory during seqscans

2022-11-23 Thread David Rowley
On Wed, 23 Nov 2022 at 21:26, sirisha chamarthi wrote: > Master > After vacuum: > latency average = 393.880 ms > > Master + 0001 + 0005 > After vacuum: > latency average = 369.591 ms Thank you for running those again. Those results make more sense. Would you mind also testing the count(*) query

Re: Hash index build performance tweak from sorting

2022-11-23 Thread David Rowley
On Fri, 18 Nov 2022 at 03:34, Tomas Vondra wrote: > I did some simple benchmark with v2 and v3, using the attached script, > which essentially just builds hash index on random data, with different > data types and maintenance_work_mem values. And what I see is this > (median of 10 runs): > So to

Re: Hash index build performance tweak from sorting

2022-11-23 Thread David Rowley
On Wed, 16 Nov 2022 at 17:33, Simon Riggs wrote: > > Thanks for the review, apologies for the delay in acting upon your comments. > > My tests show the sorted and random tests are BOTH 4.6% faster with > the v3 changes using 5-test avg, but you'll be pleased to know your > kit is about 15.5%

Bug in MERGE's test for tables with rules

2022-11-23 Thread Dean Rasheed
While playing around with rules and MERGE, I noticed that there is a bug in the way that it detects whether the target table has rules --- it uses rd_rel->relhasrules, which can be incorrect, since it might be set for a table that doesn't currently have rules, but did in the recent past. So it

Another multi-row VALUES bug

2022-11-23 Thread Dean Rasheed
I was thinking some more about the recent fix to multi-row VALUES handling in the rewriter (b8f2687fdc), and I realised that there is another bug in the way DEFAULT values are handled: In RewriteQuery(), the code assumes that in a multi-row INSERT query, the VALUES RTE will be the only thing in

Re: [PATCH] Enable using llvm jitlink as an alternative llvm jit linker of old Rtdyld.

2022-11-23 Thread David Rowley
On Wed, 23 Nov 2022 at 23:13, Alex Fan wrote: > I am new to the postgres community and apologise for resending this as the > previous one didn't include patch properly and didn't cc reviewers (maybe the > reason it has been buried in mailing list for months) Welcome to the community! I've not

Re: Fix for visibility check on 14.5 fails on tpcc with high concurrency

2022-11-23 Thread Alvaro Herrera
On 2022-Nov-23, Alvaro Herrera wrote: > I suggest that we could improve that elog() so that it includes the > members of the multixact in question, which could help us better > understand what is going on. Something like the attached. It would result in output like this: WARNING: new multixact

Re: Perform streaming logical transactions by background workers and parallel apply

2022-11-23 Thread Amit Kapila
On Tue, Nov 22, 2022 at 7:23 PM Hayato Kuroda (Fujitsu) wrote: > > > 07. proto.c - logicalrep_write_stream_abort() > > We may able to add assertions for abort_lsn and abort_time, like xid and > subxid. > If you see logicalrep_write_stream_commit(), we have an assertion for xid but not for LSN

Re: [PATCH] Enable using llvm jitlink as an alternative llvm jit linker of old Rtdyld.

2022-11-23 Thread Alex Fan
Hi, I am new to the postgres community and apologise for resending this as the previous one didn't include patch properly and didn't cc reviewers (maybe the reason it has been buried in mailing list for months) Adding to previous email, this patch exposes its own C API for creating

Re: odd buildfarm failure - "pg_ctl: control file appears to be corrupt"

2022-11-23 Thread Thomas Munro
On Wed, Nov 23, 2022 at 2:42 PM Andres Freund wrote: > The failure has to be happening in wait_for_postmaster_promote(), because the > standby2 is actually successfully promoted. I assume this is ext4. Presumably anything that reads the controlfile, like pg_ctl, pg_checksums, pg_resetwal,

Re: [PoC] Federated Authn/z with OAUTHBEARER

2022-11-23 Thread mahendrakar s
Hi, We validated on  libpq handling OAuth natively with different flows with different OIDC certified providers. Flows: Device Code, Client Credentials and Refresh Token. Providers: Microsoft, Google and Okta. Also validated with OAuth provider Github. We propose using OpenID Connect (OIDC) as

Re: Logical Replication Custom Column Expression

2022-11-23 Thread Stavros Koureas
Just one correction for the subscriber On Subscriber: test_sub=# CREATE TABLE tab(id int *pkey*, description varchar, subscription varchar *pkey*); CREATE TABLE The subscription table should have the same primary key columns as the publisher plus one more. We need to make sure that on update

Re: Logical Replication Custom Column Expression

2022-11-23 Thread Stavros Koureas
It's easy to answer this question. Imagine that in a software company who sells the product and also offers reporting solutions, the ERP tables will not have this additional column to all the tables. Now the reporting department comes and needs to consolidate all that data from different

Re: Non-decimal integer literals

2022-11-23 Thread John Naylor
On Wed, Nov 23, 2022 at 3:54 PM David Rowley wrote: > > Going by [1], clang will actually use multiplication by 16 to > implement the former. gcc is better and shifts left by 4, so likely > won't improve things for gcc. It seems worth doing it this way for > anything that does not have

Re: Fix for visibility check on 14.5 fails on tpcc with high concurrency

2022-11-23 Thread Alvaro Herrera
Hello Dimos On 2022-Nov-22, Dimos Stamatakis wrote: > When running tpcc on sysbench with high concurrency (96 threads, scale > factor 5) we realized that a fix for visibility check (introduced in > PG-14.5) causes sysbench to fail in 1 out of 70 runs. > The error is the following: > > SQL

Re: Logical replication missing information

2022-11-23 Thread Boboc Cristi
Hello!Thank you very much, the information you just sent to me is very, very valuable! Best regards, Cristi Boboc On Wednesday, November 23, 2022 at 04:45:09 AM GMT+2, Peter Smith wrote: On Fri, Nov 18, 2022 at 4:50 AM PG Doc comments form wrote: > > The following documentation

Re: odd buildfarm failure - "pg_ctl: control file appears to be corrupt"

2022-11-23 Thread Alvaro Herrera
On 2022-Nov-22, Andres Freund wrote: > ok 10 - standby is in recovery > # Running: pg_ctl -D > /mnt/resource/bf/build/grassquit/REL_11_STABLE/pgsql.build/src/bin/pg_ctl/tmp_check/t_003_promote_standby2_data/pgdata > promote > waiting for server to promotepg_ctl: control file appears to be

Re: Operation log for major operations

2022-11-23 Thread Alvaro Herrera
On 2022-Nov-21, Dmitry Koval wrote: > Concepts. > - > * operation log is placed in the file 'global/pg_control', starting from > position 4097 (log size is 4kB); I think storing this in pg_control is a bad idea. That file is extremely critical and if you break it, you're pretty much SOL

Re: Non-decimal integer literals

2022-11-23 Thread David Rowley
On Wed, 23 Nov 2022 at 21:54, David Rowley wrote: > I wonder if you'd be better off with something like: > > while (*ptr && isxdigit((unsigned char) *ptr)) > { > if (unlikely(tmp & UINT64CONST(0xF000))) > goto out_of_range; > >

Re: Non-decimal integer literals

2022-11-23 Thread David Rowley
On Wed, 23 Nov 2022 at 02:37, Peter Eisentraut wrote: > Here is a new patch. This looks like quite an inefficient way to convert a hex string into an int64: while (*ptr && isxdigit((unsigned char) *ptr)) { int8digit = hexlookup[(unsigned char) *ptr];

Re: [DOCS] Stats views and functions not in order?

2022-11-23 Thread Peter Smith
On Thu, Nov 17, 2022 at 8:46 AM David G. Johnston wrote: > > On Tue, Nov 15, 2022 at 6:39 PM Peter Smith wrote: >> >> >> I was also wondering (but have not yet done) if the content *outside* >> the tables should be reordered to match the table 28.1/28.2 order. >> >> Thoughts? >> Thanks for the

Re: Prefetch the next tuple's memory during seqscans

2022-11-23 Thread sirisha chamarthi
On Tue, Nov 22, 2022 at 11:44 PM David Rowley wrote: > On Wed, 23 Nov 2022 at 20:29, sirisha chamarthi > wrote: > > I ran your test1 exactly like your setup except the row count is 300 > (with 13275 blocks). Shared_buffers is 128MB and the hardware configuration > details at the bottom of

Re: New docs chapter on Transaction Management and related changes

2022-11-23 Thread Justin Pryzby
On Tue, Nov 22, 2022 at 01:50:36PM -0500, Bruce Momjian wrote: > + > + > + A more complex example with multiple nested subtransactions: > + > +BEGIN; > +INSERT INTO table1 VALUES (1); > +SAVEPOINT sp1; > +INSERT INTO table1 VALUES (2); > +SAVEPOINT sp2; > +INSERT INTO

Re: More efficient build farm animal wakeup?

2022-11-23 Thread Thomas Munro
On Wed, Nov 23, 2022 at 2:09 PM Andres Freund wrote: > It's a huge improvement here. Same here. eelpout + elver looking good, just a fraction of a second hitting that web server each minute. Long polling will be better and shave off 30 seconds (+/- 30) on start time, but this avoids a lot of

<    1   2