Re: First draft of the PG 15 release notes

2022-05-13 Thread Amit Kapila
On Fri, May 13, 2022 at 9:18 PM Bruce Momjian wrote: > > On Fri, May 13, 2022 at 08:24:53AM +0530, Amit Kapila wrote: > > On Fri, May 13, 2022 at 6:02 AM Euler Taveira wrote: > > > > > > On Thu, May 12, 2022, at 11:22 AM, Bruce Momjian wrote: > > > > > > On Thu, May 12, 2022 at 11:12:54AM -0300,

Re: Backends stunk in wait event IPC/MessageQueueInternal

2022-05-13 Thread Thomas Munro
On Sat, May 14, 2022 at 10:25 AM Thomas Munro wrote: > Japin, are you able to reproduce the problem reliably? Did I guess > right, that you're on illumos? Does this help? I used > defined(__sun__) to select the option, but I don't remember if that's > the right way to detect that OS family,

Re: Making JIT more granular

2022-05-13 Thread Andy Fan
Hi David: > Does anyone have any thoughts about this JIT costing? Is this an > improvement? Is there a better way? > > I think this is an improvement. However I'm not sure how much improvement & effort we want pay for it. I just shared my thoughts to start this discussion. 1. Ideally there

Re: wrong fds used for refilenodes after pg_upgrade relfilenode changes Reply-To:

2022-05-13 Thread Thomas Munro
On Sat, May 14, 2022 at 3:33 AM Robert Haas wrote: > This seems fine, but I think you should add a non-trivial comment about it. Thanks for looking. Done, and pushed. Let's see if 180s per query is enough...

Re: Correct comment in ProcedureCreate() for pgstat_create_function() call.

2022-05-13 Thread Michael Paquier
On Fri, May 13, 2022 at 04:09:00PM +0900, Michael Paquier wrote: > As of what pgstat_create_function() does to create the stats of a new > function in a transactional way, it looks like you are right. Will > fix if there are no objections. And done with fcab82a. Thanks, Amul. -- Michael

Re: [RFC] building postgres with meson -v8

2022-05-13 Thread Andres Freund
Hi, On 2022-05-11 12:18:58 +0200, Peter Eisentraut wrote: > This currently only works on macOS. The dtrace -G calls needed on > other platforms are not implemented yet. I looked into that part. The make rule passes all the backend object files as an option, but it's not clear to me where / why

Re: Removing unneeded self joins

2022-05-13 Thread Zhihong Yu
Hi, w.r.t. v33-0001-Remove-self-joins.patch : removes inner join of plane table -> removes inner join of plain table in an query plan -> in a query plan + * Used as the relation_has_unique_index_for, Since relation_has_unique_index_for() becomes a wrapper of relation_has_unique_index_ext, the

Re: Backends stunk in wait event IPC/MessageQueueInternal

2022-05-13 Thread Thomas Munro
On Sat, May 14, 2022 at 9:25 AM Thomas Munro wrote: > In short, I'd recommend -DWAIT_USE_POLL for now. It's possible that > we could do something to prevent the selection of WAIT_USE_EPOLL on > that platform, or that we should have a halfway option epoll() but not > signalfd() (= go back to

Re: Backends stunk in wait event IPC/MessageQueueInternal

2022-05-13 Thread Thomas Munro
On Sat, May 14, 2022 at 2:09 AM Robert Haas wrote: > On Fri, May 13, 2022 at 6:16 AM Japin Li wrote: > > The process cannot be terminated by pg_terminate_backend(), although > > it returns true. > One thing I find a bit curious is that the top of the stack in your > case is ioctl(). And there

Re: recovery test failure on morepork with timestamp mystery

2022-05-13 Thread Mikael Kjellström
On 2022-05-13 22:09, Andres Freund wrote: What does sysctl kern.timecounter return? Does the problem continue if you switch kern.timecounter.hardware to something else? openbsd 5.9: $ sysctl kern.timecounter kern.timecounter.tick=1 kern.timecounter.timestepwarnings=0

list of TransactionIds

2022-05-13 Thread Alvaro Herrera
We didn't have any use of TransactionId as members of List, until RelationSyncEntry->streamed_txns was introduced (464824323e57, pg14). It's currently implemented as a list of int. This is not wrong at present, but it may soon be, and I'm sure it rubs some people the wrong way. But is the

Re: recovery test failure on morepork with timestamp mystery

2022-05-13 Thread Andres Freund
Hi, On 2022-05-13 10:22:32 -0700, Andres Freund wrote: > On 2022-05-13 09:00:20 +0200, Mikael Kjellström wrote: > > Well, I don't know if you remember but there was a thread a while back and a > > test program (monotime.c) to test the clock if it could go backwards and > > openbsd showed the

Re: Item compression in the Gist index

2022-05-13 Thread Bruce Momjian
On Mon, May 2, 2022 at 02:48:27PM +0200, Baca Radim wrote: > Hi, > we are developing an extension for multidimensional data. We have created a > Gist index that is heavily inspired by a cube extension. Right now we would > like to add some item compression since data items in a node share a >

Re: Privileges on PUBLICATION

2022-05-13 Thread Euler Taveira
On Fri, May 13, 2022, at 3:36 AM, Antonin Houska wrote: > Attached is my proposal. It tries to be more specific and does not mention the > absence of the privileges explicitly. You explained the current issue but say nothing about the limitation. This information will trigger a question possibly

Re: Multi-Master Logical Replication

2022-05-13 Thread Bruce Momjian
On Fri, Apr 29, 2022 at 07:05:11PM +1000, Peter Smith wrote: > This MMLR proposal was mostly just to create an interface making it > easier to use PostgreSQL core logical replication CREATE > PUBLICATION/SUBSCRIPTION for table sharing among a set of nodes. > Otherwise, this is difficult for a user

Re: recovery test failure on morepork with timestamp mystery

2022-05-13 Thread Andres Freund
Hi, On 2022-05-13 09:00:20 +0200, Mikael Kjellström wrote: > Well, I don't know if you remember but there was a thread a while back and a > test program (monotime.c) to test the clock if it could go backwards and > openbsd showed the following result when running the attached testprogram: Nope,

Re: Comments on Custom RMGRs

2022-05-13 Thread Robert Haas
On Fri, May 13, 2022 at 8:47 AM Simon Riggs wrote: > > Note that I'm *not* against making checkpoint extensible - I just think it > > needs a good bit of design work around when the hook is called etc. > > When was any such work done previously for any other hook?? That isn't needed. I think

Re: Should use MERGE use BulkInsertState ?

2022-05-13 Thread Robert Haas
On Wed, May 11, 2022 at 12:48 PM Justin Pryzby wrote: > The point of the ring/strategy buffer is to avoid a seq scan/vacuum/COPY > clobbering the entire buffer cache when processing a table larger than > shared_buffers. > > It also makes backends doing bulk operations responsible for their own

Re: First draft of the PG 15 release notes

2022-05-13 Thread Bruce Momjian
On Fri, May 13, 2022 at 08:24:53AM +0530, Amit Kapila wrote: > On Fri, May 13, 2022 at 6:02 AM Euler Taveira wrote: > > > > On Thu, May 12, 2022, at 11:22 AM, Bruce Momjian wrote: > > > > On Thu, May 12, 2022 at 11:12:54AM -0300, Euler Taveira wrote: > > OB> On Thu, May 12, 2022, at 11:03 AM,

Re: Comments on Custom RMGRs

2022-05-13 Thread Jeff Davis
On Fri, 2022-05-13 at 13:31 +0100, Simon Riggs wrote: > The first was a discussion about a data structure needed by BDR about > 4 years ago. In the absence of a pluggable checkpoint, the solution > was forced to use a normal table, which wasn't very satisfactory. I'm interested to hear more about

Re: First draft of the PG 15 release notes

2022-05-13 Thread Bruce Momjian
On Fri, May 13, 2022 at 10:48:41AM +0900, Amit Langote wrote: > On Thu, May 12, 2022 at 10:52 PM Bruce Momjian wrote: > > Okay, I went with: > > > > Previously, such updates ran delete actions on the source > > partition and insert actions on the target partition. PostgreSQL > >

Re: make MaxBackends available in _PG_init

2022-05-13 Thread Nathan Bossart
On Fri, May 13, 2022 at 09:49:54AM -0400, Robert Haas wrote: > Committed. Thanks! -- Nathan Bossart Amazon Web Services: https://aws.amazon.com

Re: wrong fds used for refilenodes after pg_upgrade relfilenode changes Reply-To:

2022-05-13 Thread Robert Haas
On Thu, May 12, 2022 at 10:20 PM Thomas Munro wrote: > As for skink failing, the timeout was hard coded 300s for the whole > test, but apparently that wasn't enough under valgrind. Let's use the > standard PostgreSQL::Test::Utils::timeout_default (180s usually), but > reset it for each query we

Re: First draft of the PG 15 release notes

2022-05-13 Thread 'Bruce Momjian'
On Fri, May 13, 2022 at 01:36:04AM +, osumi.takami...@fujitsu.com wrote: > > > > This is enabled with the subscriber option "disable_on_error" > > and avoids possible infinite loops during stream application. > > > > > Thank you ! > > In this last paragraph, how about

RE: Data is copied twice when specifying both child and parent table in publication

2022-05-13 Thread osumi.takami...@fujitsu.com
On Friday, May 13, 2022 6:42 PM Wang, Wei/王 威 wrote: > Attach the patches.(Only changed the patch for HEAD.). > 1. Optimize the code. Reduce calls to function filter_partitions. > [suggestions by > Amit-san] 2. Improve the alias name in SQL. [suggestions by Amit-san] 3. > Improve coding

Re: Backends stunk in wait event IPC/MessageQueueInternal

2022-05-13 Thread Robert Haas
On Fri, May 13, 2022 at 6:16 AM Japin Li wrote: > The process cannot be terminated by pg_terminate_backend(), although > it returns true. pg_terminate_backend() just sends SIGINT. What I'm wondering is what happens when the stuck process receives SIGINT. It would be useful, I think, to check the

Re: make MaxBackends available in _PG_init

2022-05-13 Thread Robert Haas
On Thu, May 12, 2022 at 8:15 PM Michael Paquier wrote: > On Thu, May 12, 2022 at 06:51:59PM +0300, Anton A. Melnikov wrote: > > Maybe remove the first part from the patchset? > > Because now the Patch Tester is giving apply error for the first part and > > can't test the other. > >

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2022-05-13 Thread Maxim Orlov
> here is the rebased v32 version of the patch. > > The patchset rotted a bit. Here is a rebased version. > We have posted an updated version v34 of the whole patchset in [1]. Changes of patches 0001-0003 there are identical to v33. So, no update is needed in this thread. [1]

Re: Backends stunk in wait event IPC/MessageQueueInternal

2022-05-13 Thread Japin Li
On Fri, 13 May 2022 at 19:41, Justin Pryzby wrote: > On Fri, May 13, 2022 at 06:16:23PM +0800, Japin Li wrote: >> I had an incident on my Postgres 14 that queries hung in wait event >> IPC / MessageQueueInternal, MessageQueueReceive. It likes [1], >> however, it doesn't have any discussions. >

Re: Add WAL recovery messages with log_wal_traffic GUC (was: add recovery, backup, archive, streaming etc. activity messages to server logs along with ps display)

2022-05-13 Thread Robert Haas
On Fri, Apr 29, 2022 at 5:11 AM Bharath Rupireddy wrote: > Here's the rebased v9 patch. This seems like it has enormous overlap with the existing functionality that we have from log_startup_progress_interval. I think that facility is also better-designed than this one. It prints out a message

Re: Add WAL recovery messages with log_wal_traffic GUC (was: add recovery, backup, archive, streaming etc. activity messages to server logs along with ps display)

2022-05-13 Thread Bharath Rupireddy
On Thu, May 5, 2022 at 2:07 PM Alvaro Herrera wrote: > > On 2022-May-05, Bharath Rupireddy wrote: > > > On Fri, Apr 29, 2022 at 4:11 PM Alvaro Herrera > > wrote: > > > > > > Did we ever consider the idea of using a new pg_stat_wal_activity_progress > > > view or something like that, using the

Re: Comments on Custom RMGRs

2022-05-13 Thread Simon Riggs
On Fri, 13 May 2022 at 00:42, Andres Freund wrote: > > On 2022-05-12 22:26:51 +0100, Simon Riggs wrote: > > On Thu, 12 May 2022 at 04:40, Andres Freund wrote: > > > I'm not happy with the idea of random code being executed in the middle of > > > CheckPointGuts(), without any documentation of

Re: Can postgres ever delete the recycled future WAL files to free-up disk space if max_wal_size is reduced or wal_recycle is set to off?

2022-05-13 Thread Justin Pryzby
On Fri, May 06, 2022 at 09:50:26PM +0530, Bharath Rupireddy wrote: > It seems like that's not the case because postgres will not remove future WAL > files even after max_wal_size is reduced, In the past, I've had to generate synthetic write traffic and checkpoints to get WAL to shrink. +1 to

Re: Unfiltered server logs routing via a new elog hook or existing emit_log_hook bypassing log_min_message check

2022-05-13 Thread Bharath Rupireddy
On Mon, May 2, 2022 at 7:37 PM Tom Lane wrote: > > Julien Rouhaud writes: > > On Mon, May 02, 2022 at 07:24:04PM +0530, Bharath Rupireddy wrote: > >> I basically want to avoid normal users/developers setting any > >> parameter (especially the superuser-only log_min_message GUC, all > >> users

Re: Can postgres ever delete the recycled future WAL files to free-up disk space if max_wal_size is reduced or wal_recycle is set to off?

2022-05-13 Thread Bharath Rupireddy
On Mon, May 9, 2022 at 6:47 PM Bharath Rupireddy wrote: > > On Fri, May 6, 2022 at 10:20 PM Tom Lane wrote: > > > > Bharath Rupireddy writes: > > > Can postgres delete the recycled future WAL files once max_wal_size is > > > reduced and/or wal_recycle is set to off? > > > > A checkpoint should

Re: Comments on Custom RMGRs

2022-05-13 Thread Simon Riggs
On Fri, 13 May 2022 at 05:13, Jeff Davis wrote: > > On Thu, 2022-05-12 at 22:26 +0100, Simon Riggs wrote: > > I see multiple uses for the rm_checkpoint() point proposed and I've > > been asked multiple times for a checkpoint hook. > > Can you elaborate and/or link to a discussion? Those were

Re: An attempt to avoid locally-committed-but-not-replicated-to-standby-transactions in synchronous replication

2022-05-13 Thread Bharath Rupireddy
On Tue, May 10, 2022 at 5:55 PM Andrey Borodin wrote: > > > 10 мая 2022 г., в 12:59, Bharath Rupireddy > > написал(а): > > > > If okay, I can make the GUC behave this way - value 0 existing > > behaviour i.e. no wait for sync repl ack, just process query cancels > > and proc die interrupts

Re: Backends stunk in wait event IPC/MessageQueueInternal

2022-05-13 Thread Justin Pryzby
On Fri, May 13, 2022 at 06:16:23PM +0800, Japin Li wrote: > I had an incident on my Postgres 14 that queries hung in wait event > IPC / MessageQueueInternal, MessageQueueReceive. It likes [1], > however, it doesn't have any discussions. If the process is still running, or if the problem recurs,

Backends stunk in wait event IPC/MessageQueueInternal

2022-05-13 Thread Japin Li
Hi, hackers I had an incident on my Postgres 14 that queries hung in wait event IPC / MessageQueueInternal, MessageQueueReceive. It likes [1], however, it doesn't have any discussions. The process cannot be terminated by pg_terminate_backend(), although it returns true. Here is the call

RE: Data is copied twice when specifying both child and parent table in publication

2022-05-13 Thread wangw.f...@fujitsu.com
On Fri, May 13, 2022 1:59 PM Amit Kapila wrote: > On Fri, May 13, 2022 at 7:32 AM wangw.f...@fujitsu.com > wrote: > > > > Attach the patches.(Only changed the patch for HEAD.). > > > > Few comments: > = Thanks for your comments. > 1. > @@ -1135,6 +1172,15 @@

Re: postgres_fdw "parallel_commit" docs

2022-05-13 Thread Etsuro Fujita
Hi Jonathan, On Thu, May 12, 2022 at 10:32 PM Jonathan S. Katz wrote: > On 5/12/22 7:26 AM, Etsuro Fujita wrote: > > Attached is an updated patch. I'll commit the patch if no objections. > > I think this is much easier to read. Cool! > I made a few minor copy edits. Please see attached.

RE: Perform streaming logical transactions by background workers and parallel apply

2022-05-13 Thread houzj.f...@fujitsu.com
On Wednesday, May 11, 2022 1:10 PM Amit Kapila wrote: > > On Wed, May 11, 2022 at 9:35 AM Masahiko Sawada > wrote: > > > > On Tue, May 10, 2022 at 5:59 PM Amit Kapila > wrote: > > > > > > On Tue, May 10, 2022 at 10:39 AM Masahiko Sawada > wrote: > > > > > > > > Having it optional seems a good

RE: Perform streaming logical transactions by background workers and parallel apply

2022-05-13 Thread houzj.f...@fujitsu.com
On Thursday, May 5, 2022 1:46 PM Peter Smith wrote: > Here are my review comments for v5-0001. > I will take a look at the v5-0002 (TAP) patch another time. Thanks for the comments ! > 4. Commit message > > User can set the streaming option to 'on/off', 'apply'. For now, > 'apply' means the

Re: [PATCH] Compression dictionaries for JSONB

2022-05-13 Thread Aleksander Alekseev
Hi hackers, > Here it the 2nd version of the patch: > > - Includes changes named above > - Fixes a warning reported by cfbot > - Fixes some FIXME's > - The path includes some simple tests now > - A proper commit message was added Here is the rebased version of the patch. Changes compared to v2

Re: Correct comment in ProcedureCreate() for pgstat_create_function() call.

2022-05-13 Thread Michael Paquier
On Fri, May 13, 2022 at 10:22:57AM +0530, Amul Sul wrote: > Sorry, hit the send button too early :| - /* ensure that stats are dropped if transaction commits */ + /* ensure that stats are dropped if transaction aborts */ if (!is_update) pgstat_create_function(retval); As of

Re: recovery test failure on morepork with timestamp mystery

2022-05-13 Thread Mikael Kjellström
On 2022-05-13 04:14, Andres Freund wrote: One interesting tidbit is that the log timestamps are computed differently (with elog.c:get_formatted_log_time()) than the reset timestamp (with GetCurrentTimestamp()). Both use gettimeofday() internally. I wonder if there's a chance that somehow

Re: Privileges on PUBLICATION

2022-05-13 Thread Antonin Houska
Peter Eisentraut wrote: > On 10.05.22 10:37, Antonin Houska wrote: > > My understanding is that the rows/columns filtering is a way for the > > *publisher* to control which data is available to particular replica. From > > this point of view, the publication privileges would just make the

Re: Privileges on PUBLICATION

2022-05-13 Thread Antonin Houska
Euler Taveira wrote: > On Tue, May 10, 2022, at 5:37 AM, Antonin Houska wrote: > > My understanding is that the rows/columns filtering is a way for the > *publisher* to control which data is available to particular replica. From > this point of view, the publication privileges would just

RE: bogus: logical replication rows/cols combinations

2022-05-13 Thread houzj.f...@fujitsu.com
On Thursday, May 12, 2022 2:45 PM Amit Kapila wrote: > > On Wed, May 11, 2022 at 12:55 PM houzj.f...@fujitsu.com > wrote: > > > > On Wednesday, May 11, 2022 11:33 AM Amit Kapila > wrote: > > > > > > Fair enough, then we should go with a simpler approach to detect it > > > in pgoutput.c