Re: Using per-transaction memory contexts for storing decoded tuples

2024-10-03 Thread Masahiko Sawada
On Thu, Oct 3, 2024 at 2:46 AM Fujii Masao wrote: > > > > On 2024/10/03 13:47, Masahiko Sawada wrote: > >>> I agree that the overhead will be much less visible in real workloads. > >>> +1 to use a smaller block (i.e. 8kB). > > +1 > > > >>&g

Re: Enhance file_fdw to report processed and skipped tuples in COPY progress

2024-10-03 Thread Masahiko Sawada
s://www.postgresql.org/message-id/flat/20230119054703.GB13860%40telsasoft.com Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: pg_trgm comparison bug on cross-architecture replication due to different char implementation

2024-10-03 Thread Masahiko Sawada
On Wed, Oct 2, 2024 at 10:02 AM Masahiko Sawada wrote: > > On Tue, Oct 1, 2024 at 8:57 PM Tom Lane wrote: > > > > Noah Misch writes: > > > On Tue, Oct 01, 2024 at 11:55:48AM -0700, Masahiko Sawada wrote: > > >> Considering that the population of database

Re: Using per-transaction memory contexts for storing decoded tuples

2024-10-02 Thread Masahiko Sawada
ms that > only reorderbuffer.c uses the LARGE macro so that it can be removed. I'm going to keep the LARGE macro since extensions might be using it. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Add on_error and log_verbosity options to file_fdw

2024-10-02 Thread Masahiko Sawada
On Tue, Oct 1, 2024 at 11:34 PM Fujii Masao wrote: > > > > On 2024/10/02 9:27, Masahiko Sawada wrote: > > Sorry for being late in joining the review of this patch. Both 0001 > > and 0003 look good to me. I have two comments on the 0002 patch: > > Thanks for the re

Re: pg_trgm comparison bug on cross-architecture replication due to different char implementation

2024-10-02 Thread Masahiko Sawada
On Tue, Oct 1, 2024 at 8:57 PM Tom Lane wrote: > > Noah Misch writes: > > On Tue, Oct 01, 2024 at 11:55:48AM -0700, Masahiko Sawada wrote: > >> Considering that the population of database cluster signedness will > >> converge to signedness=true in the future, we ca

Re: Add on_error and log_verbosity options to file_fdw

2024-10-01 Thread Masahiko Sawada
every time before parsing an input line. Would it be problematic if we switch to another memory context while parsing an input line? In CopyFrom() we also call ResetPerTupleExprContext() and ExecClearTuple() for every input, so we might want to consider calling them for every input. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Using per-transaction memory contexts for storing decoded tuples

2024-10-01 Thread Masahiko Sawada
On Tue, Oct 1, 2024 at 5:15 AM Amit Kapila wrote: > > On Fri, Sep 27, 2024 at 10:24 PM Masahiko Sawada > wrote: > > > > On Fri, Sep 27, 2024 at 12:39 AM Shlok Kyal > > wrote: > > > > > > On Mon, 23 Sept 2024 at 09:59, Amit Kapila > > > w

Re: pg_trgm comparison bug on cross-architecture replication due to different char implementation

2024-10-01 Thread Masahiko Sawada
On Mon, Sep 30, 2024 at 11:49 AM Noah Misch wrote: > > On Wed, Sep 25, 2024 at 03:46:27PM -0700, Masahiko Sawada wrote: > > On Wed, Sep 18, 2024 at 6:46 PM Noah Misch wrote: > > > On Tue, Sep 17, 2024 at 09:43:41PM -0700, Masahiko Sawada wrote: > > > > On Mon, S

Re: Make COPY format extendable: Extract COPY TO format implementations

2024-09-27 Thread Masahiko Sawada
both of the FROM and TO part? > * Could someone provide a hint why the FROM part is more > slower with Ryzen? > Separating the patches into two parts (one is for COPY TO and another one is for COPY FROM) could be a good idea. It would help reviews and investigate performance regression in COPY FROM cases. And I think we can commit them separately. Also, could you please rebase the patches as they conflict with the current HEAD? I'll run some benchmarks on my environment as well. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Vacuum statistics

2024-09-27 Thread Masahiko Sawada
On Fri, Sep 27, 2024 at 12:19 PM Melanie Plageman wrote: > > On Fri, Sep 27, 2024 at 2:16 PM Masahiko Sawada wrote: > > > > Hi, > > > > On Thu, Sep 5, 2024 at 2:01 PM Alena Rybakina > > wrote: > > > > > > Hi! Thank you for your

Re: Vacuum statistics

2024-09-27 Thread Masahiko Sawada
t. I think it would be better to avoid collecting duplicated statistics in different places. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Using per-transaction memory contexts for storing decoded tuples

2024-09-27 Thread Masahiko Sawada
12395.8364 ms |76.78273689 > 8MB |11712.8862 ms |50.74323039 > ====== > > From the results, I think there is a small regression for small block size. > > I ran the tests in git bash. I have also attached the test script. Thank you for testing on Windows! I've run the same benchmark on Mac (Sonoma 14.7, M1 Pro): 8kB: 4852.198 ms 16kB: 4822.733 ms 32kB: 4776.776 ms 64kB: 4851.433 ms 128kB: 4804.821 ms 256kB: 4781.778 ms 512kB: 4776.486 ms 1MB: 4783.456 ms 2MB: 4770.671 ms 4MB: 4785.800 ms 8MB: 4747.447 ms I can see there is a small regression for small block sizes. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Pgoutput not capturing the generated columns

2024-09-26 Thread Masahiko Sawada
: specified generated column "c" in publication column list for publication with publish_generated_columns as false Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Using per-transaction memory contexts for storing decoded tuples

2024-09-25 Thread Masahiko Sawada
s the reason why the small block test does > > not cause much of a performance regression. I wonder how the small > > block test would look on Mac, FreeBSD or Windows. I think it would be > > risky to assume that all is well with reducing the block size after > > testing on a single platform. > > > > Good point. We need extensive testing on different platforms, as you > suggest, to verify if smaller block sizes caused any regressions. +1. I'll do the same test on my Mac as well. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Add contrib/pg_logicalsnapinspect

2024-09-25 Thread Masahiko Sawada
[SNAPBUILD_CONSISTENT + SNAPBUILD_STATE_INCR] = "consistent", +}; I think that it'd be better to have a dedicated function that returns a string representation of the given state by using a switch statement. That way, we don't need SNAPBUILD_STATE_INCR and a compiler warning would help realize a missing state if a new state is introduced in the future. It needs a function call but I believe it won't be a problem in this use case. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: pg_trgm comparison bug on cross-architecture replication due to different char implementation

2024-09-25 Thread Masahiko Sawada
On Wed, Sep 18, 2024 at 6:46 PM Noah Misch wrote: > > On Tue, Sep 17, 2024 at 09:43:41PM -0700, Masahiko Sawada wrote: > > On Mon, Sep 16, 2024 at 9:24 AM Noah Misch wrote: > > > On Thu, Sep 12, 2024 at 03:42:48PM -0700, Masahiko Sawada wrote: > > > > On Tue, S

Re: Conflict detection for update_deleted in logical replication

2024-09-24 Thread Masahiko Sawada
flush location using the existing logical replication connection (i.e., between the logical wal sender and the apply worker), and advertise the locations on the shared memory. Then, the central process who holds the slot to retain the deleted row versions traverses them and increases slot.xmin if possible. The cost of requesting the remote wal flush location would not be huge if we don't ask it very frequently. So probably we can start by having each apply worker (in the retain_sub_list) ask the remote wal flush location and can leave the optimization of avoiding sending the request for the same publisher. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Conflict detection for update_deleted in logical replication

2024-09-24 Thread Masahiko Sawada
On Tue, Sep 24, 2024 at 12:14 AM Zhijie Hou (Fujitsu) wrote: > > On Tuesday, September 24, 2024 2:42 PM Masahiko Sawada > wrote: > > > > On Mon, Sep 23, 2024 at 8:32 PM Zhijie Hou (Fujitsu) > > wrote: > > > > > > On Tuesday, September 24, 2024 5:

Re: Conflict detection for update_deleted in logical replication

2024-09-23 Thread Masahiko Sawada
On Mon, Sep 23, 2024 at 8:32 PM Zhijie Hou (Fujitsu) wrote: > > On Tuesday, September 24, 2024 5:05 AM Masahiko Sawada > wrote: > > > > Thank you for considering another idea. > > Thanks for reviewing the idea! > > > > > On Fri, Sep 20, 2024 at 2:46

Re: Using per-transaction memory contexts for storing decoded tuples

2024-09-23 Thread Masahiko Sawada
On Thu, Sep 19, 2024 at 10:44 PM Amit Kapila wrote: > > On Thu, Sep 19, 2024 at 10:33 PM Masahiko Sawada > wrote: > > > > On Wed, Sep 18, 2024 at 8:55 PM Amit Kapila wrote: > > > > > > On Thu, Sep 19, 2024 at 6:46 AM David Rowley wrote: > > > &

Re: Conflict detection for update_deleted in logical replication

2024-09-23 Thread Masahiko Sawada
didate_xmin and > candidate_remote_wal_lsn) in slot. We can probably reuse existing > candidate variables of the slot. Next, we can check the remote_flush > locations from all the origins corresponding in retain_sub_list and if > all are ahead of candidate_remote_wal_lsn, we can update the slot's > xmin to candidate_xmin. Does it mean that we use one candiate_remote_wal_lsn in a slot for all subscriptions (in retain_sub_list)? IIUC candiate_remote_wal_lsn is a LSN of one of publishers, so other publishers could have completely different LSNs. How do we compare the candidate_remote_wal_lsn to remote_flush locations from all the origins? Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Pgoutput not capturing the generated columns

2024-09-20 Thread Masahiko Sawada
On Thu, Sep 19, 2024 at 9:26 PM Amit Kapila wrote: > > On Fri, Sep 20, 2024 at 4:16 AM Peter Smith wrote: > > > > On Fri, Sep 20, 2024 at 3:26 AM Masahiko Sawada > > wrote: > > > > > > On Thu, Sep 19, 2024 at 2:32 AM Amit Kapila > >

Re: Using per-transaction memory contexts for storing decoded tuples

2024-09-20 Thread Masahiko Sawada
On Thu, Sep 19, 2024 at 10:46 PM Amit Kapila wrote: > > On Fri, Sep 20, 2024 at 5:13 AM David Rowley wrote: > > > > On Fri, 20 Sept 2024 at 05:03, Masahiko Sawada > > wrote: > > > I've done other benchmarking tests while changing the memory block >

Re: Conflict detection for update_deleted in logical replication

2024-09-19 Thread Masahiko Sawada
On Tue, Sep 17, 2024 at 9:29 PM Amit Kapila wrote: > > On Tue, Sep 17, 2024 at 11:24 PM Masahiko Sawada > wrote: > > > > On Mon, Sep 16, 2024 at 11:53 PM Amit Kapila > > wrote: > > > > > > On Tue, Sep 17, 2024 at 6:08 AM Masahiko Sawada > &g

Re: Pgoutput not capturing the generated columns

2024-09-19 Thread Masahiko Sawada
On Thu, Sep 19, 2024 at 2:32 AM Amit Kapila wrote: > > On Tue, Sep 17, 2024 at 12:04 PM Peter Smith wrote: > > > > On Tue, Sep 17, 2024 at 4:15 PM Masahiko Sawada > > wrote: > > > > > > On Mon, Sep 16, 2024 at 8:09 PM Peter Smith wrote:

Re: Using per-transaction memory contexts for storing decoded tuples

2024-09-19 Thread Masahiko Sawada
appreciated if you also do similar tests and share the results. Regards, [1] https://www.postgresql.org/message-id/CAD21AoAaN4jaJ%3DW%2BWprHvc0cGCf80SkiFQhRc6R%2BX3-05HAFqw%40mail.gmail.com -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Using per-transaction memory contexts for storing decoded tuples

2024-09-19 Thread Masahiko Sawada
On Wed, Sep 18, 2024 at 8:55 PM Amit Kapila wrote: > > On Thu, Sep 19, 2024 at 6:46 AM David Rowley wrote: > > > > On Thu, 19 Sept 2024 at 11:54, Masahiko Sawada > > wrote: > > > I've done some benchmark tests for three different code bases with > >

Re: Using per-transaction memory contexts for storing decoded tuples

2024-09-18 Thread Masahiko Sawada
On Tue, Sep 17, 2024 at 2:06 AM Amit Kapila wrote: > > On Mon, Sep 16, 2024 at 10:43 PM Masahiko Sawada > wrote: > > > > On Fri, Sep 13, 2024 at 3:58 AM Amit Kapila wrote: > > > > > > Can we try reducing the size of > > > 8MB memory bl

Re: pg_trgm comparison bug on cross-architecture replication due to different char implementation

2024-09-17 Thread Masahiko Sawada
On Mon, Sep 16, 2024 at 9:24 AM Noah Misch wrote: > > On Thu, Sep 12, 2024 at 03:42:48PM -0700, Masahiko Sawada wrote: > > On Tue, Sep 10, 2024 at 3:05 PM Noah Misch wrote: > > > On Tue, Sep 10, 2024 at 05:56:47PM -0400, Tom Lane wrote: > > > > Got it. So now

Re: Using per-transaction memory contexts for storing decoded tuples

2024-09-17 Thread Masahiko Sawada
On Tue, Sep 17, 2024 at 2:06 AM Amit Kapila wrote: > > On Mon, Sep 16, 2024 at 10:43 PM Masahiko Sawada > wrote: > > > > On Fri, Sep 13, 2024 at 3:58 AM Amit Kapila wrote: > > > > > > On Thu, Sep 12, 2024 at 4:03 AM Masahiko Sawada > > > wro

Re: Conflict detection for update_deleted in logical replication

2024-09-17 Thread Masahiko Sawada
On Mon, Sep 16, 2024 at 11:53 PM Amit Kapila wrote: > > On Tue, Sep 17, 2024 at 6:08 AM Masahiko Sawada wrote: > > > > On Fri, Sep 13, 2024 at 12:56 AM shveta malik > > wrote: > > > > > > On Fri, Sep 13, 2024 at 11:38 AM Amit Kapila > > >

Re: Pgoutput not capturing the generated columns

2024-09-16 Thread Masahiko Sawada
an error (as we do today)? or always > > send NULL? > > For this scenario, I suggested (see [1] #3) that the code could give a > WARNING. As I wrote up-thread: This combination doesn't seem > like something a user would do intentionally, so just silently > ignoring it (which the current patch does) is likely going to give > someone unexpected results/grief. It gives a WARNING, and then publishes the specified generated column data (even if publish_generated_column = false)? If so, it would mean that specifying the generated column to the column list means to publish its data regardless of the publish_generated_column parameter value. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Conflict detection for update_deleted in logical replication

2024-09-16 Thread Masahiko Sawada
nt to > handle replication delays caused by network lag or other factors, > assuming clock skew has already been addressed. I think that in a non-bidirectional case the value could need to be a large number. Is that right? Regards, [1] https://www.postgresql.org/message-id/20230317230930.nhsgk3qfk7f4axls%40awork3.anarazel.de -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Pgoutput not capturing the generated columns

2024-09-16 Thread Masahiko Sawada
users specify a generated column to the column list and set publish_generated_column = false, in the first place? raise an error (as we do today)? or always send NULL? Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Using per-transaction memory contexts for storing decoded tuples

2024-09-16 Thread Masahiko Sawada
On Fri, Sep 13, 2024 at 3:58 AM Amit Kapila wrote: > > On Thu, Sep 12, 2024 at 4:03 AM Masahiko Sawada wrote: > > > > We have several reports that logical decoding uses memory much more > > than logical_decoding_work_mem[1][2][3]. For instance in one of the > > rep

Re: pg_trgm comparison bug on cross-architecture replication due to different char implementation

2024-09-12 Thread Masahiko Sawada
On Tue, Sep 10, 2024 at 3:05 PM Noah Misch wrote: > > On Tue, Sep 10, 2024 at 05:56:47PM -0400, Tom Lane wrote: > > Masahiko Sawada writes: > > > On Tue, Sep 10, 2024 at 11:57 AM Tom Lane wrote: > > >> Yeah, that seems like it could work. But are we sure that

Using per-transaction memory contexts for storing decoded tuples

2024-09-11 Thread Masahiko Sawada
ad. Regards, [1] https://www.postgresql.org/message-id/CAMnUB3oYugXCBLSkih%2BqNsWQPciEwos6g_AMbnz_peNoxfHwyw%40mail.gmail.com [2] https://www.postgresql.org/message-id/17974-f8c9d353a62f414d%40postgresql.org [3] https://www.postgresql.org/message-id/DB9PR07MB71808AC6C7770AF2FB36B95BCB252%40DB9

Re: pg_trgm comparison bug on cross-architecture replication due to different char implementation

2024-09-10 Thread Masahiko Sawada
On Tue, Sep 10, 2024 at 11:57 AM Tom Lane wrote: > > Masahiko Sawada writes: > > An alternative way would be that we store the char signedness in the > > control file, and gin_trgm_ops opclass reads it if the bytes in the > > meta page shows 'unset'. The c

Re: pg_trgm comparison bug on cross-architecture replication due to different char implementation

2024-09-10 Thread Masahiko Sawada
On Mon, Sep 9, 2024 at 11:25 PM Tom Lane wrote: > > Masahiko Sawada writes: > > On Mon, Sep 9, 2024 at 4:42 PM Tom Lane wrote: > >> Do you have an idea for how we'd get > >> this to happen during pg_upgrade, exactly? > > > What I was thinking is th

Re: First draft of PG 17 release notes

2024-09-10 Thread Masahiko Sawada
ser that is already mentioned in the release note would fall in this type too; it's not a feature aimed just for extension authors, but it's kind of source and internal changes IMO. Since the DSM registry feature is described in the doc, I think it would make sense to have it in the release no

Re: pg_trgm comparison bug on cross-architecture replication due to different char implementation

2024-09-09 Thread Masahiko Sawada
On Mon, Sep 9, 2024 at 4:42 PM Tom Lane wrote: > > Masahiko Sawada writes: > > When do we set the byte on the primary server? If it's the first time > > to use the GIN index, secondary servers would have to wait for the > > primary to use the GIN index, which coul

Re: pg_trgm comparison bug on cross-architecture replication due to different char implementation

2024-09-09 Thread Masahiko Sawada
would have to wait for the primary to use the GIN index, which could be an unpredictable time or it may never come depending on index usages. Probably we can make pg_upgrade set the byte in the meta page of GIN indexes that use the gin_trgm_ops. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Pgoutput not capturing the generated columns

2024-09-09 Thread Masahiko Sawada
On Mon, Sep 9, 2024 at 2:38 AM Shubham Khanna wrote: > > On Thu, Aug 29, 2024 at 11:46 AM Amit Kapila wrote: > > > > On Thu, Aug 29, 2024 at 8:44 AM Masahiko Sawada > > wrote: > > > > > > On Wed, Aug 28, 2024 at 1:06 AM Amit Kapila > > > wro

Re: pg_trgm comparison bug on cross-architecture replication due to different char implementation

2024-09-06 Thread Masahiko Sawada
On Fri, Aug 30, 2024 at 8:10 PM Noah Misch wrote: > > On Thu, Aug 29, 2024 at 03:48:53PM -0500, Masahiko Sawada wrote: > > On Sun, May 19, 2024 at 6:46 AM Noah Misch wrote: > > > If I were standardizing pg_trgm on one or the other notion of "char", I > > &g

Re: pg_trgm comparison bug on cross-architecture replication due to different char implementation

2024-08-29 Thread Masahiko Sawada
On Sun, May 19, 2024 at 6:46 AM Noah Misch wrote: > > On Wed, May 15, 2024 at 02:56:54PM +0900, Masahiko Sawada wrote: > > > > How about extending amcheck to support GIN and GIst indexes so that it > > can detect potential data incompatibility due to changing 'ch

Re: Pgoutput not capturing the generated columns

2024-08-28 Thread Masahiko Sawada
On Wed, Aug 28, 2024 at 1:06 AM Amit Kapila wrote: > > On Mon, May 20, 2024 at 1:49 PM Masahiko Sawada wrote: > > > > On Wed, May 8, 2024 at 4:14 PM Shubham Khanna > > wrote: > > > > > > On Wed, May 8, 2024 at 11:39 AM Rajendra Kumar Dangwal &

Re: Fix memory counter update in reorderbuffer

2024-08-26 Thread Masahiko Sawada
On Mon, Aug 26, 2024 at 2:27 PM Nathan Bossart wrote: > > On Mon, Aug 26, 2024 at 11:25:53AM -0700, Masahiko Sawada wrote: > > Thank you for reviewing the patch. Pushed. > > nitpick: I think this one needs a pgindent run. Thank you for pointing it out. I forgot to check w

Re: Fix memory counter update in reorderbuffer

2024-08-26 Thread Masahiko Sawada
On Sun, Aug 25, 2024 at 9:29 PM Amit Kapila wrote: > > On Fri, Aug 23, 2024 at 3:44 PM Masahiko Sawada wrote: > > > > On Tue, Aug 20, 2024 at 9:29 PM Amit Kapila wrote: > > > > > > > I've updated the updated patch with regression tests. > > &g

Re: Fix memory counter update in reorderbuffer

2024-08-23 Thread Masahiko Sawada
On Tue, Aug 20, 2024 at 9:29 PM Amit Kapila wrote: > > On Tue, Aug 20, 2024 at 5:55 PM Masahiko Sawada wrote: > > > > On Fri, Aug 16, 2024 at 12:22 AM Shlok Kyal > > wrote: > > > > > > > Thank you for testing the patch! > > > > I'

Re: Fix memory counter update in reorderbuffer

2024-08-20 Thread Masahiko Sawada
Hi, On Fri, Aug 16, 2024 at 12:22 AM Shlok Kyal wrote: > > On Wed, 7 Aug 2024 at 11:48, Amit Kapila wrote: > > > > On Wed, Aug 7, 2024 at 7:42 AM Masahiko Sawada > > wrote: > > > > > > On Tue, Aug 6, 2024 at 1:12 PM Amit Kapila > > > wr

Re: Use pgBufferUsage for block reporting in analyze

2024-08-13 Thread Masahiko Sawada
On Fri, Aug 2, 2024 at 8:11 AM Masahiko Sawada wrote: > > On Wed, Jul 31, 2024 at 11:27 PM Anthonin Bonnefoy > wrote: > > > > On Wed, Jul 31, 2024 at 9:36 PM Masahiko Sawada > > wrote: > > > Meanwhile, I think we can push 0001 and 0002 patches since they a

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-08-12 Thread Masahiko Sawada
validation when replaying a RUNNING_XACTS record. Since a RUNNING_XACTS record has the latest XID on the primary, I think the startup process can compare it to the slot-xmin, and invalidate slots which are older than the age limit. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Fix memory counter update in reorderbuffer

2024-08-11 Thread Masahiko Sawada
On Sat, Aug 10, 2024 at 5:48 AM Masahiko Sawada wrote: > > On Fri, Aug 9, 2024 at 3:30 PM Amit Kapila wrote: > > > > On Thu, Aug 8, 2024 at 9:43 PM Masahiko Sawada > > wrote: > > > > > > On Wed, Aug 7, 2024 at 3:17 PM Amit Kapila > > > wr

Re: Fix memory counter update in reorderbuffer

2024-08-09 Thread Masahiko Sawada
On Fri, Aug 9, 2024 at 3:30 PM Amit Kapila wrote: > > On Thu, Aug 8, 2024 at 9:43 PM Masahiko Sawada wrote: > > > > On Wed, Aug 7, 2024 at 3:17 PM Amit Kapila wrote: > > > > > > On Wed, Aug 7, 2024 at 7:42 AM Masahiko Sawada > > > wrote: > &

Re: Fix memory counter update in reorderbuffer

2024-08-08 Thread Masahiko Sawada
On Wed, Aug 7, 2024 at 3:17 PM Amit Kapila wrote: > > On Wed, Aug 7, 2024 at 7:42 AM Masahiko Sawada wrote: > > > > On Tue, Aug 6, 2024 at 1:12 PM Amit Kapila wrote: > > > > > > On Sat, Aug 3, 2024 at 1:21 AM Masahiko Sawada > > > wrote: > &

Re: Fix memory counter update in reorderbuffer

2024-08-06 Thread Masahiko Sawada
On Tue, Aug 6, 2024 at 1:12 PM Amit Kapila wrote: > > On Sat, Aug 3, 2024 at 1:21 AM Masahiko Sawada wrote: > > > > I found a bug in the memory counter update in reorderbuffer. It was > > introduced by commit 5bec1d6bc5e, so pg17 and master are affected. > > >

Fix memory counter update in reorderbuffer

2024-08-02 Thread Masahiko Sawada
the patch that fixes the problem and includes a test case (the test part might not be committed as it slows the test case). Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com fix_memory_counter_update_in_reorderbuffer.patch Description: Binary data

Re: Use pgBufferUsage for block reporting in analyze

2024-08-01 Thread Masahiko Sawada
On Wed, Jul 31, 2024 at 11:27 PM Anthonin Bonnefoy wrote: > > On Wed, Jul 31, 2024 at 9:36 PM Masahiko Sawada wrote: > > Meanwhile, I think we can push 0001 and 0002 patches since they are in > > good shape. I've updated commit messages to them and slightly changed

Re: Use pgBufferUsage for block reporting in analyze

2024-07-31 Thread Masahiko Sawada
they are in good shape. I've updated commit messages to them and slightly changed 0002 patch to write "finished analyzing of table \"%s.%s.%s\" instead of "analyze of table \"%s.%s.%s\". Also, regarding 0003 patch, what is the main reason why we want to add WAL

Re: xid_wraparound tests intermittent failure.

2024-07-30 Thread Masahiko Sawada
On Tue, Jul 30, 2024 at 3:29 AM Andrew Dunstan wrote: > > > On 2024-07-29 Mo 5:25 PM, Masahiko Sawada wrote: > > I've attached the patch. Could you please test if the patch fixes the > instability you observed? > > Since we turn off autovacuum on all three tests an

Re: long-standing data loss bug in initial sync of logical replication

2024-07-30 Thread Masahiko Sawada
are not sure if that is good so the other alternative we > can pursue is to distribute invalidations in logical decoding > infrastructure [1] which has its downsides. > > Thoughts? Thank you for summarizing the problem and solutions! I think it's worth trying the idea of distributing invalidation messages, and we will see if there could be overheads or any further obstacles. IIUC this approach would resolve another issue we discussed before too[1]. Regards, [1] https://www.postgresql.org/message-id/CAD21AoAenVqiMjpN-PvGHL1N9DWnHSq673bfgr6phmBUzx=k...@mail.gmail.com -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Use pgBufferUsage for block reporting in analyze

2024-07-29 Thread Masahiko Sawada
Hi, On Mon, Jul 29, 2024 at 12:12 AM Anthonin Bonnefoy wrote: > > On Sat, Jul 27, 2024 at 12:35 AM Masahiko Sawada > wrote: > > An alternative idea would > > be to pass StringInfo to AcquireSampleRowsFunc() so that callback can > > write its messages there. This is

Re: xid_wraparound tests intermittent failure.

2024-07-29 Thread Masahiko Sawada
On Sat, Jul 27, 2024 at 1:06 PM Andrew Dunstan wrote: > > > On 2024-07-26 Fr 1:46 PM, Masahiko Sawada wrote: > > On Thu, Jul 25, 2024 at 6:52 PM Andrew Dunstan wrote: > >> > >> On 2024-07-25 Th 3:40 PM, Masahiko Sawada wrote: > >> > >&g

Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin

2024-07-28 Thread Masahiko Sawada
On Fri, Jul 26, 2024 at 1:27 PM Melanie Plageman wrote: > > On Mon, Jul 22, 2024 at 9:26 PM Masahiko Sawada wrote: > > > > + CREATE TABLE ${table1}(col1 int) > > + WITH (autovacuum_enabled=false, fillfactor=10); > > + INSERT INTO $table1 VALUES(7); >

Re: Use pgBufferUsage for block reporting in analyze

2024-07-26 Thread Masahiko Sawada
On Wed, Jul 24, 2024 at 1:58 AM Anthonin Bonnefoy wrote: > > On Mon, Jul 22, 2024 at 10:59 PM Masahiko Sawada > wrote: > > The first line would vary depending on whether an autovacuum worker or > > not. And the above suggestion includes a change of term "row&

Re: Parallel heap vacuum

2024-07-26 Thread Masahiko Sawada
ber of parallel workers as we don't want to launch many workers on the table where most pages are all-visible. Which might not work for other table AMs. I'm updating the patch to implement parallel heap vacuum and will share the updated patch. It might take time as it requires to implement shared iteration support in radx tree. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: xid_wraparound tests intermittent failure.

2024-07-26 Thread Masahiko Sawada
On Thu, Jul 25, 2024 at 6:52 PM Andrew Dunstan wrote: > > > On 2024-07-25 Th 3:40 PM, Masahiko Sawada wrote: > > On Thu, Jul 25, 2024 at 11:06 AM Masahiko Sawada > wrote: > > On Thu, Jul 25, 2024 at 10:56 AM Andrew Dunstan wrote: > > On 2024-07-23 Tu 6:59 PM,

Re: Detailed release notes

2024-07-26 Thread Masahiko Sawada
changes, and I believe many users would like to do that. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: xid_wraparound tests intermittent failure.

2024-07-25 Thread Masahiko Sawada
On Thu, Jul 25, 2024 at 11:06 AM Masahiko Sawada wrote: > > On Thu, Jul 25, 2024 at 10:56 AM Andrew Dunstan wrote: > > > > > > On 2024-07-23 Tu 6:59 PM, Masahiko Sawada wrote: > > > > See > > <https://bitbucket.org/adunstan/rotfang-fdw/downloads/x

Re: xid_wraparound tests intermittent failure.

2024-07-25 Thread Masahiko Sawada
On Thu, Jul 25, 2024 at 10:56 AM Andrew Dunstan wrote: > > > On 2024-07-23 Tu 6:59 PM, Masahiko Sawada wrote: > > See > <https://bitbucket.org/adunstan/rotfang-fdw/downloads/xid-wraparound-result.tar.bz2> > > > The failure logs are from a run where both tests

Re: xid_wraparound tests intermittent failure.

2024-07-23 Thread Masahiko Sawada
On Tue, Jul 23, 2024 at 3:49 AM Andrew Dunstan wrote: > > > On 2024-07-22 Mo 9:29 PM, Masahiko Sawada wrote: > > On Mon, Jul 22, 2024 at 12:53 PM Andrew Dunstan wrote: > > On 2024-07-22 Mo 12:46 PM, Tom Lane wrote: > > Masahiko Sawada writes: > > Looking at dod

Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin

2024-07-23 Thread Masahiko Sawada
On Tue, Jul 23, 2024 at 5:43 AM Melanie Plageman wrote: > > On Mon, Jul 22, 2024 at 10:54 PM Masahiko Sawada > wrote: > > > > On Mon, Jul 22, 2024 at 6:26 PM Melanie Plageman > > wrote: > > > > > > On Mon, Jul 22, 2024 at 6:36 PM Tom Lane wr

Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin

2024-07-22 Thread Masahiko Sawada
-- regardless > of how slowly it runs vacuum. Looking at copperhead's failure logs, I could not find that "VACUUM (VERBOSE, FREEZE) vac_horizon_floor_table;" wrote the number of index scans in logs. Is there any clue that made you think the test failed to do multiple index vacuum passes? Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: xid_wraparound tests intermittent failure.

2024-07-22 Thread Masahiko Sawada
On Mon, Jul 22, 2024 at 12:53 PM Andrew Dunstan wrote: > > > On 2024-07-22 Mo 12:46 PM, Tom Lane wrote: > > Masahiko Sawada writes: > > Looking at dodo's failures, it seems that while it passes > module-xid_wraparound-check, all failures happened only during > tes

Re: Vacuum ERRORs out considering freezing dead tuples from before OldestXmin

2024-07-22 Thread Masahiko Sawada
urrent TidStore, we will have to re-adjust them if the TidStore changes in the future. So I think it's better and reliable to allow maintenance_work_mem to be a lower value or use injection points somehow. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Add on_error and log_verbosity options to file_fdw

2024-07-22 Thread Masahiko Sawada
FOREIGN TABLE > > > > =# table f_fdw_test_3 ; > > NOTICE: skipping row due to data type incompatibility at line 3 for > > column i: "a" > > NOTICE: 1 row was skipped due to data type incompatibility > >i | t > > ---+--- > >1 | a > >2 | b > > (2 rows) IIUC we have to execute ALTER FOREIGN TABLE to change the log_verbosity value and which requires to be the owner. Which seems not to be user-friendly. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Use pgBufferUsage for block reporting in analyze

2024-07-22 Thread Masahiko Sawada
row" to "tuple" for better consistency with VACUUM VERBOSE outputs. I think it would be great if autoanalyze also writes logs in the same format. IIUC with the patch, autoanalyze logs don't include the page and tuple statistics. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: xid_wraparound tests intermittent failure.

2024-07-22 Thread Masahiko Sawada
On Mon, Jul 22, 2024 at 9:46 AM Tom Lane wrote: > > Masahiko Sawada writes: > > Looking at dodo's failures, it seems that while it passes > > module-xid_wraparound-check, all failures happened only during > > testmodules-install-check-C. Can we check the s

Re: xid_wraparound tests intermittent failure.

2024-07-22 Thread Masahiko Sawada
test, "slow disk" could also be a reason. Looking at dodo's failures, it seems that while it passes module-xid_wraparound-check, all failures happened only during testmodules-install-check-C. Can we check the server logs written during xid_wraparound test in testmodules-install-check-C? I t

Re: MERGE/SPLIT partition commands should create new partitions in the parent's tablespace?

2024-07-12 Thread Masahiko Sawada
On Thu, Jul 11, 2024 at 8:14 PM Fujii Masao wrote: > > > > On 2024/07/10 22:35, Masahiko Sawada wrote: > > BTW the new regression tests don't check the table and index names. > > Isn't it better to show table and index names for better > > diagnosability?

Re: MERGE/SPLIT partition commands should create new partitions in the parent's tablespace?

2024-07-10 Thread Masahiko Sawada
On Wed, Jul 10, 2024 at 5:14 PM Masahiko Sawada wrote: > > On Wed, Jul 10, 2024 at 4:14 PM Fujii Masao > wrote: > > > > > > > > On 2024/07/10 12:13, Masahiko Sawada wrote: > > > On Sat, Jul 6, 2024 at 4:06 PM Fujii Masao > > > wrote: > &g

Re: MERGE/SPLIT partition commands should create new partitions in the parent's tablespace?

2024-07-10 Thread Masahiko Sawada
On Wed, Jul 10, 2024 at 4:14 PM Fujii Masao wrote: > > > > On 2024/07/10 12:13, Masahiko Sawada wrote: > > On Sat, Jul 6, 2024 at 4:06 PM Fujii Masao > > wrote: > >> > >> Hi, > >> > >> I noticed that ALTER TABLE MERGE PARTITIONS and S

Re: MERGE/SPLIT partition commands should create new partitions in the parent's tablespace?

2024-07-09 Thread Masahiko Sawada
le's tablespace, it is logical to set the parent table's tablespace as the merged table's tablespace. While the patch does not include test cases for SPLIT PARTITIONS, which is understandable as these commands use the common function that we have fixed, I believe it would be prudent to test SPLIT PARTITIONS as well since we could change it in the future development. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Parallel heap vacuum

2024-07-07 Thread Masahiko Sawada
On Fri, Jun 28, 2024 at 9:06 PM Amit Kapila wrote: > > On Fri, Jun 28, 2024 at 9:44 AM Masahiko Sawada wrote: > > > > # Benchmark results > > > > * Test-1: parallel heap scan on the table without indexes > > > > I created 20GB table, made garbage on th

Re: Parallel heap vacuum

2024-07-07 Thread Masahiko Sawada
attached the new version patch. BTW since we compute the number of parallel workers for the heap scan based on the table size, it's possible that we launch multiple workers even if most blocks are all-visible. It seems to be better if we calculate it based on (relpages - relallvisible). Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com parallel_heap_vacuum_scan_v2.patch Description: Binary data

Re: Conflict Detection and Resolution

2024-07-03 Thread Masahiko Sawada
On Mon, Jul 1, 2024 at 6:54 PM Amit Kapila wrote: > > On Mon, Jul 1, 2024 at 1:35 PM Masahiko Sawada wrote: > > > > Setting resolvers at table-level and subscription-level sounds good to > > me. DDLs for setting resolvers at subscription-level would need the > > sub

Re: Conflict Detection and Resolution

2024-07-01 Thread Masahiko Sawada
eason we are planning to keep resolvers at the > > table level. Here, I am asking to set resolvers at the subscription > > level rather than at the global level. > > Okay, got it. I misunderstood earlier that we want to replace table > level resolvers with subscription ones. > Having global configuration has one benefit that if the user has no > requirement to set different resolvers for different subscriptions or > tables, he may always set one global configuration and be done with > it. OTOH, I also agree with benefits coming with subscription level > configuration. Setting resolvers at table-level and subscription-level sounds good to me. DDLs for setting resolvers at subscription-level would need the subscription name to be specified? And another question is: a table-level resolver setting is precedent over all subscriber-level resolver settings in the database? Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: Conflict Detection and Resolution

2024-06-30 Thread Masahiko Sawada
because the commit timestamp of UPDATE is newer than DELETE' one.ard DELETE, because the commit timestamp of UPDATE is newer than DELETE' one. As a result, both nodes have the new version row. Regards, [1] https://www.enterprisedb.com/docs/pgd/latest/consistency/conflicts/#updatedelete-conflicts [2] https://docs.oracle.com/goldengate/c1230/gg-winux/GWUAD/configuring-conflict-detection-and-resolution.htm (see DELETEROWEXISTS conflict type) -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Parallel heap vacuum

2024-06-27 Thread Masahiko Sawada
e ideas but it might be applicable to our ART implementation. But I prefer to start with (2) since it would be easier. Feedback is very welcome. Regards, [1] https://db.in.tum.de/~leis/papers/artsync.pdf -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com diff --git a/src/backend/access/he

Re: Track the amount of time waiting due to cost_delay

2024-06-26 Thread Masahiko Sawada
add any extra pg_clock_gettime_ns() calls (as compare to v2). > Sounds good to me. I think it's better to keep the logic for throttling the reporting the delay message simple. It's an important consideration but executing parallel vacuum with delays would be less likely to be used in practice. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: New standby_slot_names GUC in PG 17

2024-06-26 Thread Masahiko Sawada
On Wed, Jun 26, 2024 at 6:15 PM Zhijie Hou (Fujitsu) wrote: > > On Wednesday, June 26, 2024 12:49 PM Bertrand Drouvot > wrote: > > > > Hi, > > > > On Wed, Jun 26, 2024 at 04:17:45AM +, Zhijie Hou (Fujitsu) wrote: > > > On Wednesday, June 2

Re: Vacuum statistics

2024-06-26 Thread Masahiko Sawada
is hard to predict, I think showing the high-water mark would help users to predict how much memory they set to maintenance_work_mem. Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com

Re: New standby_slot_names GUC in PG 17

2024-06-25 Thread Masahiko Sawada
On Tue, Jun 25, 2024 at 5:32 PM Amit Kapila wrote: > > On Tue, Jun 25, 2024 at 12:30 PM Masahiko Sawada > wrote: > > > > On Tue, Jun 25, 2024 at 1:54 PM Amit Kapila wrote: > > > > > > > > So, my > > > preference is in order as follows: sy

Re: New standby_slot_names GUC in PG 17

2024-06-25 Thread Masahiko Sawada
On Tue, Jun 25, 2024 at 1:54 PM Amit Kapila wrote: > > On Tue, Jun 25, 2024 at 8:20 AM Masahiko Sawada wrote: > > > > On Tue, Jun 25, 2024 at 11:21 AM Zhijie Hou (Fujitsu) > > wrote: > > > > > > I agree the current name seems too generic and the sug

Re: New standby_slot_names GUC in PG 17

2024-06-24 Thread Masahiko Sawada
lots_on_standbys: it indicates that the standbys that enabled > slot sync should be listed in this GUC. > > logical_replication_wait_slots: it means the logical replication(logical > Walsender process) will wait for these slots to advance the confirm flush > lsn before proceeding. I fee

Re: suspicious valgrind reports about radixtree/tidstore on arm64

2024-06-20 Thread Masahiko Sawada
On Thu, Jun 20, 2024 at 4:58 PM John Naylor wrote: > > On Thu, Jun 20, 2024 at 8:12 AM Masahiko Sawada wrote: > > > On Thu, Jun 20, 2024 at 7:54 AM Tom Lane wrote: > > > > > > I don't know if there's any reason why the current order > > >

Re: suspicious valgrind reports about radixtree/tidstore on arm64

2024-06-19 Thread Masahiko Sawada
fix-proposed.patch does. I agree with radixtree-fix-proposed.patch. Even if we zero more fields in the node it would not add noticeable overheads. > > (The RT_NODE_48 case could be optimized a bit if we cared > to swap the order of its slot_idxs[] and isset[] arrays; > then the initial

Re: Revive num_dead_tuples column of pg_stat_progress_vacuum

2024-06-18 Thread Masahiko Sawada
On Sat, Jun 15, 2024 at 8:47 PM Robert Treat wrote: > > On Thu, Jun 13, 2024 at 9:22 PM Masahiko Sawada wrote: > > On Fri, Jun 14, 2024 at 9:57 AM Masahiko Sawada > > wrote: > > > On Fri, Jun 14, 2024 at 9:41 AM Michael Paquier > > > wrote: > > &g

Re: Logical Replication of sequences

2024-06-17 Thread Masahiko Sawada
On Fri, Jun 14, 2024 at 4:04 PM Amit Kapila wrote: > > On Thu, Jun 13, 2024 at 6:14 PM Masahiko Sawada wrote: > > > > On Thu, Jun 13, 2024 at 7:06 PM Amit Kapila wrote: > > > > > > On Thu, Jun 13, 2024 at 1:09 PM Masahiko Sawada > > > wrote: &g

  1   2   3   4   5   6   7   8   9   10   >