Re: Single transaction in the tablesync worker?

On Mon, Dec 7, 2020 at 10:02 AM Craig Ringer wrote: > > On Mon, 7 Dec 2020 at 11:44, Peter Smith wrote: >> >> >> Basically, I was wondering why can't the "tablesync" worker just >> gather messages in a similar way to how the current streaming feature >> gathers messages into a "changes" file, so

RE: [Patch] Optimize dropping of relation buffers using dlist

On Friday, December 4, 2020 8:27 PM, Amit Kapila wrote: > On Fri, Nov 27, 2020 at 11:36 AM Kyotaro Horiguchi > wrote: > > > > At Fri, 27 Nov 2020 02:19:57 +, "k.jami...@fujitsu.com" > > wrote in > > > > From: Kyotaro Horiguchi Hello, Kirk. > > > > Thank you for the new version. > > > > >

Re: ModifyTable overheads in generic plans

On Thu, Nov 12, 2020 at 5:04 PM Amit Langote wrote: > Attached new 0002 which does these adjustments. I went with > ri_RootTargetDesc to go along with ri_RelationDesc. > > Also, I have updated the original 0002 (now 0003) to make > GetChildToRootMap() use ri_RootTargetDesc instead of >

Re: POC: Better infrastructure for automated testing of concurrency issues

On Wed, 25 Nov 2020 at 22:11, Alexander Korotkov wrote: > Hackers, > > PostgreSQL is a complex multi-process system, and we are periodically > faced with complicated concurrency issues. While the postgres community > does a great job on investigating and fixing the problems, our ability to >

RE: [bug fix] ALTER TABLE SET LOGGED/UNLOGGED on a partitioned table does nothing silently

From: Bharath Rupireddy > IMHO, we should also change the parent table. Say, I have 2 local > partitions for a logged table, then I alter that table to > unlogged(with your patch, parent table doesn't become unlogged whereas > the partitions will), and I detach all the partitions for some reason.

RE: Parallel Inserts in CREATE TABLE AS

Hi + /* +* Flag to let the planner know that the SELECT query is for CTAS. This is +* used to calculate the tuple transfer cost from workers to gather node(in +* case parallelism kicks in for the SELECT part of the CTAS), to zero as +* each worker will

Re: Improving spin-lock implementation on ARM.

On Sat, 5 Dec 2020 at 02:55, Alexander Korotkov wrote: > > On Wed, Dec 2, 2020 at 6:58 AM Krunal Bauskar wrote: > > Let me know what do you think about this analysis and any specific > > direction that we should consider to help move forward. > > BTW, it would be also nice to benchmark my

Re: Parallel Inserts in CREATE TABLE AS

Thanks for the comments. On Mon, Dec 7, 2020 at 8:56 AM Zhihong Yu wrote: > > > + (void) SetCurrentCommandIdUsedForWorker(); > + myState->output_cid = GetCurrentCommandId(false); > > SetCurrentCommandIdUsedForWorker already has void as return type. The > '(void)' is not needed. >

Re: Proposed patch for key managment

On Mon, Dec 7, 2020 at 11:03:30AM +0900, Michael Paquier wrote: > On Sat, Dec 05, 2020 at 10:42:05AM -0500, Bruce Momjian wrote: > > On Sat, Dec 5, 2020 at 09:54:33PM +0900, Michael Paquier wrote: > >> On Fri, Dec 04, 2020 at 10:52:29PM -0500, Bruce Momjian wrote: > >>> if (state->evpctx ==

Re: Proposed patch for key managment

On Mon, Dec 7, 2020 at 09:30:03AM +0900, Masahiko Sawada wrote: > Thank you for updating the patch! > > I think we need explicit_bzero() also in freeing the keywrap context. pg_cryptohash_free() already has this: explicit_bzero(state, sizeof(pg_cryptohash_state)); explicit_bzero(ctx,

Re: Single transaction in the tablesync worker?

On Mon, 7 Dec 2020 at 11:44, Peter Smith wrote: > > Basically, I was wondering why can't the "tablesync" worker just > gather messages in a similar way to how the current streaming feature > gathers messages into a "changes" file, so that they can be replayed > later. > > See the related thread

RFC: Deadlock detector hooks for edge injection

Hi all Related to my other post about deadlock detector hooks for victim selection, I'd like to gauge opinions here about whether it's feasible to inject edges into the deadlock detector's waits-for graph. Doing so would help with detecting deadlocks relating to shm_mq waits, help with

RFC: Deadlock detector hooks for victim selection and edge injection

Hi folks Now that we're well on track for streaming logical decoding, it's becoming more practical to look at parallel logical apply. The support for this in pglogical3 benefits from a deadlock detector hook. It was added in the optional patched postgres pglogical can use to enable various extra

Re: Single transaction in the tablesync worker?

On Mon, Dec 7, 2020 at 6:20 AM Craig Ringer wrote: > > On Sat, 5 Dec 2020, 10:03 Amit Kapila, wrote: >> >> On Fri, Dec 4, 2020 at 7:12 PM Ashutosh Bapat >> wrote: >> >> I think the problem is not that the changes are visible after COPY >> rather it is that we don't have a mechanism to restart

Re: Single transaction in the tablesync worker?

Hi, I wanted to float another idea to solve these tablesync/apply worker problems. This idea may or may not have merit. Please consider it. ~ Basically, I was wondering why can't the "tablesync" worker just gather messages in a similar way to how the current streaming feature gathers messages

RFC: Giving bgworkers walsender-like grace during shutdown (for logical replication)

Hi folks TL;DR: Anyone object to a new bgworker flag that exempts background workers (such as logical apply workers) from the first round of fast shutdown signals, and instead lets them to finish their currently in-progress txn and exit? This is a change proposal raised for comment before patch

Re: Parallel Inserts in CREATE TABLE AS

Hi, Bharath : + (void) SetCurrentCommandIdUsedForWorker(); + myState->output_cid = GetCurrentCommandId(false); SetCurrentCommandIdUsedForWorker already has void as return type. The '(void)' is not needed. +* rd_createSubid is marked invalid, otherwise, the table is +

Re: Logical archiving

Actually CC'd Petr this time. On Mon, 7 Dec 2020 at 11:05, Craig Ringer wrote: > Reply follows inline. I addressed your last point first, so it's out of > order. > > On Fri, 4 Dec 2020 at 15:33, Andrey Borodin wrote > > > If OLAP cannot consume data fast enough - we are out of space due to >

Re: Logical archiving

Reply follows inline. I addressed your last point first, so it's out of order. On Fri, 4 Dec 2020 at 15:33, Andrey Borodin wrote > If OLAP cannot consume data fast enough - we are out of space due to repl slot. There is a much simpler solution to this than logical PITR. What we should be

Re: A few new options for CHECKPOINT

On Sun, Dec 06, 2020 at 10:03:08AM -0500, Stephen Frost wrote: > * Alvaro Herrera (alvhe...@alvh.no-ip.org) wrote: >> You keep making this statement, and I don't necessarily disagree, but if >> that is the case, please explain why don't we have >> checkpoint_completion_target set to 0.9 by

Re: Proposed patch for key managment

On Sat, Dec 05, 2020 at 10:42:05AM -0500, Bruce Momjian wrote: > On Sat, Dec 5, 2020 at 09:54:33PM +0900, Michael Paquier wrote: >> On Fri, Dec 04, 2020 at 10:52:29PM -0500, Bruce Momjian wrote: >>> if (state->evpctx == NULL) >>> { >>> - explicit_bzero(state,

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

> + /* Make a guess at a good size when we're not given a valid size. */ > + if (size == 0) > + size = 1024; > > Should the default size be logged ? > I'm not too sure if I know what you mean here. Should it be a power of > 2? It is. Or do you mean I shouldn't use a magic number?

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

On Sat, 5 Dec 2020 at 16:51, Zhihong Yu wrote: > > There are two blocks with almost identical code (second occurrence in > cache_store_tuple): > > + if (rcstate->mem_used > rcstate->mem_upperlimit) > + { > It would be nice if the code can be extracted to a method and shared. It's true,

Re: Single transaction in the tablesync worker?

On Sat, 5 Dec 2020, 10:03 Amit Kapila, wrote: > On Fri, Dec 4, 2020 at 7:12 PM Ashutosh Bapat > wrote: > > > > On Thu, Dec 3, 2020 at 7:24 PM Amit Kapila > wrote: > > > > > > On Thu, Dec 3, 2020 at 7:04 PM Ashutosh Bapat > > > wrote: > > > > > > > > On Thu, Dec 3, 2020 at 2:55 PM Amit Kapila

Re: Parallel Inserts in CREATE TABLE AS

Thanks Amit for the review comments. On Sat, Dec 5, 2020 at 4:27 PM Amit Kapila wrote: > > > If I'm not wrong, I think currently we have no exec nodes for DDLs. > > I'm not sure whether we would like to introduce one for this. > > Yeah, I am also not in favor of having an executor node for CTAS

Re: Proposed patch for key managment

On Sun, 6 Dec 2020 at 00:42, Bruce Momjian wrote: > > On Sat, Dec 5, 2020 at 09:54:33PM +0900, Michael Paquier wrote: > > On Fri, Dec 04, 2020 at 10:52:29PM -0500, Bruce Momjian wrote: > > > OK, I worked with Sawada-san and added the attached patch. The updated > > > full patch is at the same

Re: Consider parallel for lateral subqueries with limit

> Note that near the end of grouping planner we have a similar check: > > if (final_rel->consider_parallel && root->query_level > 1 && > !limit_needed(parse)) > > guarding copying the partial paths from the current rel to the final > rel. I haven't managed to come up with a test case that

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

Thanks for having a look at this. On Sat, 5 Dec 2020 at 14:08, Zhihong Yu wrote: > +#define SH_EQUAL(tb, a, b) ResultCacheHash_equal(tb, a, b) == 0 > > I think it would be safer if the comparison is enclosed in parentheses (in > case the macro appears in composite condition). That seems fair.

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

On Sun, 6 Dec 2020 at 03:52, Andy Fan wrote: > > On Fri, Dec 4, 2020 at 10:41 PM David Rowley wrote: >> >> I also >> noticed that the code I'd written to build the cache lookup expression >> included a step to deform the outer tuple. This was unnecessary and >> slowed down the expression

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

Thanks for this review. I somehow missed addressing what's mentioned here for the v10 patch. Comments below. On Mon, 23 Nov 2020 at 02:21, Andy Fan wrote: > 1. modified src/include/utils/selfuncs.h > @@ -70,9 +70,9 @@ > * callers to provide further details about some assumptions which were

Re: Default role -> Predefined role

> On 20 Nov 2020, at 22:13, Stephen Frost wrote: > Attached is a patch to move from 'default role' terminology to > 'predefined role' in the documentation. In the code, I figured it made > more sense to avoid saying either one and instead opted for just > 'ROLE_NAME_OF_ROLE' as the #define,

Wrong check in pg_visibility?

Hi hackers! Due to the error in PG-ProEE we have added the following test to pg_visibility: create table vacuum_test as select 42 i; vacuum vacuum_test; select count(*) > 0 from pg_check_visible('vacuum_test'); drop table vacuum_test; Sometime (very rarely) this test failed: pg_visibility

pg_upgrade test for binary compatibility of core data types

I'm finally returning to this 14 month old thread: (was: Re: BUG #16045: vacuum_db crash and illegal memory alloc after pg_upgrade from PG11 to PG12) On Tue, Oct 15, 2019 at 09:07:25AM +0200, Tomas Vondra wrote: > On Mon, Oct 14, 2019 at 11:41:18PM -0500, Justin Pryzby wrote: > > > > Perhaps

Re: Change definitions of bitmap flags to bit-shifting style

On 12/6/20 11:44 AM, James Coleman wrote: > On Sun, Dec 6, 2020 at 1:25 AM Michael Paquier > wrote: > > On Sat, Dec 05, 2020 at 10:31:09PM -0300, Alvaro Herrera wrote: > > The hexadecimal representation is more natural to me than > bit-shifting, > >

Re: Change definitions of bitmap flags to bit-shifting style

On Sun, Dec 6, 2020 at 1:25 AM Michael Paquier wrote: > On Sat, Dec 05, 2020 at 10:31:09PM -0300, Alvaro Herrera wrote: > > The hexadecimal representation is more natural to me than bit-shifting, > > so I would prefer to use that style too. But maybe I'm trained to it > > because of looking at

Re: A few new options for CHECKPOINT

Greetings, * Alvaro Herrera (alvhe...@alvh.no-ip.org) wrote: > On 2020-Dec-05, Stephen Frost wrote: > > So- just to be clear, CHECKPOINTs are more-or-less always happening in > > PG, and running this command might do something or might end up doing > > nothing depending on if a checkpoint is