Re: New committers: Melanie Plageman, Richard Guo

2024-04-26 Thread Dilip Kumar
ss and few reverts! Congratulations to both of you. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Why don't we support external input/output functions for the composite types

2024-04-24 Thread Dilip Kumar
On Thu, Apr 25, 2024 at 10:14 AM Tom Lane wrote: > > Dilip Kumar writes: > > I'm curious about composite types in PostgreSQL. By default, when we > > create a composite type, it utilizes the "record_in" and "record_out" > > functions for input

Why don't we support external input/output functions for the composite types

2024-04-24 Thread Dilip Kumar
pecific input/output syntax rather than conforming to what is accepted by the RECORD type. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: SLRU optimization - configurable buffer pool and partitioning the SLRU lock

2024-04-03 Thread Dilip Kumar
quiring the bank lock w.r.t. intervening pages, thanks for reporting. Your fix looks correct to me. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: BitmapHeapScan streaming read user and prelim refactoring

2024-03-14 Thread Dilip Kumar
revious patch with some extra > assertions. Maybe for now we can just handle this specific case to remove the snapshot serializing for the BitmapHeapScan as you are doing in the patch. After looking into the code your theory seems correct that we are just copying the ActiveSnapsho

Re: BitmapHeapScan streaming read user and prelim refactoring

2024-03-13 Thread Dilip Kumar
On Wed, Mar 13, 2024 at 9:25 PM Robert Haas wrote: > > On Wed, Mar 13, 2024 at 11:39 AM Dilip Kumar wrote: > > > Andres already commented on the snapshot stuff on an earlier patch > > > version, and that's much nicer with this version. However, I don't > > > unde

Re: BitmapHeapScan streaming read user and prelim refactoring

2024-03-13 Thread Dilip Kumar
I checked other low scan nodes like indexscan and seqscan and it seems we are doing the same things there as well. Check for SerializeSnapshot() in table_parallelscan_initialize() and index_parallelscan_initialize() which are being called from ExecSeqScanInitializeDSM() and ExecIndexScanInitializeDSM

Re: Streaming I/O, vectored I/O (WIP)

2024-03-12 Thread Dilip Kumar
nice > way to solve that, but I thought this way was the simplest (and if > there is some instruction-cache-locality/tight-loop/perf reason why we > should work harder to find ranges of hits, it could be for later). > Does that make sense? Understood, I think this makes sense. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Streaming I/O, vectored I/O (WIP)

2024-03-12 Thread Dilip Kumar
d pass that blockNum into the smgrprefetch() as a first block right? -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: PostgreSQL Contributors Updates

2024-03-04 Thread Dilip Kumar
son > * Dean Rasheed > * John Naylor > * Melanie Plageman > * Nathan Bossart > > Thank you and congratulations to all! > Congratulations to all! -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: pgsql: Improve performance of subsystems on top of SLRU

2024-03-03 Thread Dilip Kumar
ainst 'MultiXactMemberCtl' related bank lock. This makes code simpler because now in the loop we are sure that we are always holding the lock but I do not like comparing the bank locks for 2 different SLRUs, although there is no problem as there would not be a common lock address, anyway, I do not have any strong objection to what you have done here. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: SLRU optimization - configurable buffer pool and partitioning the SLRU lock

2024-02-27 Thread Dilip Kumar
s 16 16 - total buffers 88 112 so that is < 200kB of extra memory on a minimally configured system, IMHO this should not matter. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: SLRU optimization - configurable buffer pool and partitioning the SLRU lock

2024-02-27 Thread Dilip Kumar
On Mon, Feb 26, 2024 at 9:46 PM Alvaro Herrera wrote: > On 2024-Feb-23, Dilip Kumar wrote: > > + > + For each SLRU area that's part of the core server, > + there is a configuration parameter that controls its size, with the > suffix > + _buffers appended. For his

Re: SLRU optimization - configurable buffer pool and partitioning the SLRU lock

2024-02-23 Thread Dilip Kumar
On Fri, Feb 23, 2024 at 1:06 PM Dilip Kumar wrote: > > On Fri, Feb 23, 2024 at 1:48 AM Alvaro Herrera > wrote: > > > > On 2024-Feb-07, Dilip Kumar wrote: > > > > > On Wed, Feb 7, 2024 at 3:49 PM Alvaro Herrera > > > wrote: > > > > >

Re: SLRU optimization - configurable buffer pool and partitioning the SLRU lock

2024-02-22 Thread Dilip Kumar
On Fri, Feb 23, 2024 at 1:48 AM Alvaro Herrera wrote: > > On 2024-Feb-07, Dilip Kumar wrote: > > > On Wed, Feb 7, 2024 at 3:49 PM Alvaro Herrera > > wrote: > > > > Sure, but is that really what we want? > > > > So your question is do we want these bu

Re: logical decoding and replication of sequences, take 2

2024-02-21 Thread Dilip Kumar
On Wed, Feb 21, 2024 at 2:52 PM Robert Haas wrote: > > On Wed, Feb 21, 2024 at 2:43 PM Dilip Kumar wrote: > > So the problem is that we might consider the transaction change as > > non-transaction and mark this flag as true. > > But it's not "might" right? It

Re: logical decoding and replication of sequences, take 2

2024-02-21 Thread Dilip Kumar
On Wed, Feb 21, 2024 at 1:24 PM Robert Haas wrote: > > On Wed, Feb 21, 2024 at 1:06 PM Dilip Kumar wrote: > > > But I am wondering why this flag is always set to true in > > > DecodeTXNNeedSkip() irrespective of the commit or abort. Because the > > > abor

Re: logical decoding and replication of sequences, take 2

2024-02-20 Thread Dilip Kumar
On Tue, Feb 20, 2024 at 4:32 PM Dilip Kumar wrote: > > On Tue, Feb 20, 2024 at 3:38 PM Robert Haas wrote: > > > Let's say fast_forward is true. Then smgr_decode() is going to skip > > recording anything about the relfilenode, so we'll identify all > > sequence c

Re: logical decoding and replication of sequences, take 2

2024-02-20 Thread Dilip Kumar
e be a situation where we have identified this flag wrongly as non-transaction and set this flag, and the commit/abort record never appeared in the WAL so never decoded? That can also lead to an incorrect decision during the upgrade. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: logical decoding and replication of sequences, take 2

2024-02-20 Thread Dilip Kumar
[2] where we concluded why this is not creating a real problem. [1] https://www.postgresql.org/message-id/CAFiTN-vAx-Y%2B19ROKOcWnGf7ix2VOTUebpzteaGw9XQyCAeK6g%40mail.gmail.com [2] https://www.postgresql.org/message-id/CAFiTN-sYpyUBabxopJysqH3DAp4OZUCTi6m_qtgt8d32vDcWSA%40mail.gmail.com -- Regards, Dil

Re: Synchronizing slots from primary to standby

2024-02-07 Thread Dilip Kumar
and anyway I > think we can even extend it after v1 is done. > Okay, I am fine with that. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: SLRU optimization - configurable buffer pool and partitioning the SLRU lock

2024-02-07 Thread Dilip Kumar
On Wed, Feb 7, 2024 at 3:49 PM Alvaro Herrera wrote: > > On 2024-Feb-07, Dilip Kumar wrote: > > > On Tue, Feb 6, 2024 at 8:55 PM Alvaro Herrera > > wrote: > > > > > > I made CLOGShmemBuffers, CommitTsShmemBuffers and SUBTRANSShmemBuffers > > > co

Re: SLRU optimization - configurable buffer pool and partitioning the SLRU lock

2024-02-06 Thread Dilip Kumar
nse for the user that it stores transaction-related data in some buffers pool but whether that buffer pool is called SLRU or not doesn't matter much to the user IMHO. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: SLRU optimization - configurable buffer pool and partitioning the SLRU lock

2024-02-06 Thread Dilip Kumar
sewhere in SLRU handling, so we don't care if we read a slightly > * outdated value; therefore we don't add a memory barrier. > */ > > Pushed with those changes. Thank you! Yeah, this looks perfect, thanks. > Now I'll go rebase the rest of the patch on top. Okay, I will review and test after that. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Synchronizing slots from primary to standby

2024-02-06 Thread Dilip Kumar
On Tue, Feb 6, 2024 at 3:41 PM Amit Kapila wrote: > > On Tue, Feb 6, 2024 at 3:23 PM Dilip Kumar wrote: > > > > On Tue, Feb 6, 2024 at 1:09 PM Masahiko Sawada > > wrote: > > > > > > On Tue, Feb 6, 2024 at 3:19 PM Amit Kapila > > > wr

Re: Question on LWLockMode in dsa.c

2024-02-06 Thread Dilip Kumar
ke to find out the correct lock mode there. > Doesn't seem like there is any reason for this to be an exclusive lock. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Synchronizing slots from primary to standby

2024-02-06 Thread Dilip Kumar
looking at this API pg_sync_replication_slots(PG_FUNCTION_ARGS) shouldn't there be an optional parameter to give one slot or multiple slots or all slots as default, that will give better control to the user no? -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Introduce XID age and inactive timeout based replication slot invalidation

2024-02-06 Thread Dilip Kumar
0), + Int32GetDatum(-1)); What is the purpose of doing this? I mean inactive_count is 8 byte integer and you can define function outparameter as 'int8' which is 8 byte integer. Then you don't need to convert int to string and then to numeric? -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: SLRU optimization - configurable buffer pool and partitioning the SLRU lock

2024-02-04 Thread Dilip Kumar
On Sun, Feb 4, 2024 at 7:10 PM Alvaro Herrera wrote: > > On 2024-Feb-02, Dilip Kumar wrote: > > > I have checked the patch and it looks fine to me other than the above > > question related to memory barrier usage one more question about the > > same, basically belo

Re: SLRU optimization - configurable buffer pool and partitioning the SLRU lock

2024-02-01 Thread Dilip Kumar
On Thu, Feb 1, 2024 at 4:34 PM Dilip Kumar wrote: > > On Thu, Feb 1, 2024 at 4:12 PM Dilip Kumar wrote: > > > > On Thu, Feb 1, 2024 at 3:44 PM Alvaro Herrera > > wrote: > > > Okay. > > > > > > While I have your attention -- if you could gi

Re: SLRU optimization - configurable buffer pool and partitioning the SLRU lock

2024-02-01 Thread Dilip Kumar
On Thu, Feb 1, 2024 at 4:12 PM Dilip Kumar wrote: > > On Thu, Feb 1, 2024 at 3:44 PM Alvaro Herrera wrote: > Okay. > > > > While I have your attention -- if you could give a look to the 0001 > > patch I posted, I would appreciate it. > > > > I

Re: SLRU optimization - configurable buffer pool and partitioning the SLRU lock

2024-02-01 Thread Dilip Kumar
On Thu, Feb 1, 2024 at 3:44 PM Alvaro Herrera wrote: > > On 2024-Feb-01, Dilip Kumar wrote: > > > On Thu, Feb 1, 2024 at 3:19 PM Alvaro Herrera > > wrote: > > > > > > postgres -c lc_messages=C -c shared_buffers=$((512*17)) > > > > > > 20

Re: SLRU optimization - configurable buffer pool and partitioning the SLRU lock

2024-02-01 Thread Dilip Kumar
10:48:13.548 CET [1535379] DETAIL: "transaction_buffers" must be > a multiple of 16 Maybe we should resize it to the next multiple of the SLRU_BANK_SIZE instead of giving an error? -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: SLRU optimization - configurable buffer pool and partitioning the SLRU lock

2024-01-28 Thread Dilip Kumar
estamp data, you deserve to wait for a few > seconds just as a punishment for your stupidity. I think this idea looks reasonable. I agree that if we are trying to read commit_ts after deactivating it then it's fine to make it wait. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: SLRU optimization - configurable buffer pool and partitioning the SLRU lock

2024-01-28 Thread Dilip Kumar
the min values were changed to 16 but in this refactoring patch for some of the buffers you have changed that to 8 so I think that's not good. + { + {"multixact_offsets_buffers", PGC_POSTMASTER, RESOURCES_MEM, + gettext_noop("Sets the size of the dedicated buffer pool used for the Mu

Re: index prefetching

2024-01-25 Thread Dilip Kumar
ry is maintained at the backend level, but what if multiple backends are trying to fetch the same heap blocks maybe scanning the same index, so should that be in some shared structure? I haven't thought much deeper about this from the implementation POV, but should we think about it, or it doesn't matter? -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Opportunistically pruning page before update

2024-01-22 Thread Dilip Kumar
e this patch? I could not find such existing pruning so got this question because such pruning can convert many non-hot updates to the HOT update right? -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] make async slave to wait for lsn to be replayed

2024-01-21 Thread Dilip Kumar
uggestion is to add some documentation in v4 and also make it more readable w.r.t. What are the clear differences between these two approaches, maybe adding commit message will also help. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Synchronizing slots from primary to standby

2024-01-19 Thread Dilip Kumar
? > Considering all this it seems that for now probably extending > replication commands can simplify a few things like mentioned above > but using SQL's with db-connection is more extendable. Even I have similar thoughts. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: btree: implement dynamic prefix truncation (was: Improving btree performance through specializing by key shape, take 2)

2024-01-18 Thread Dilip Kumar
the scan key. + */ + *compareattr = ntupatts + 1; Can you elaborate on this more? If all tuple attributes are equal to the scan key then what do those 'later attributes' mean? -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Synchronizing slots from primary to standby

2024-01-15 Thread Dilip Kumar
maintain by ourselves if we opt for option 2. I would have preferred option 3 from the simplicity point of view but I would prefer to make this PGC_SIGHUP over simplicity. But anyway, if there are issues in doing so then we can keep it PGC_POSTMASTER but it's worth trying this out. -- Regard

Re: Synchronizing slots from primary to standby

2024-01-11 Thread Dilip Kumar
ecPtr) { } if (updated_xmin) LogicalConfirmReceivedLocation(slot->data.confirmed_flush); } -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Synchronizing slots from primary to standby

2024-01-09 Thread Dilip Kumar
xmin and + * catalog_xmin values on the standby. I do not like capitalizing the first letter of the 'hot_standby_feedback' which is a GUC parameter -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: SLRU optimization - configurable buffer pool and partitioning the SLRU lock

2024-01-08 Thread Dilip Kumar
roblem with multiple processes getting the lock? I mean they are modifying different CLOG pages so that can be done concurrently right? -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Synchronizing slots from primary to standby

2024-01-07 Thread Dilip Kumar
On Fri, Jan 5, 2024 at 5:45 PM Amit Kapila wrote: > > On Fri, Jan 5, 2024 at 4:25 PM Dilip Kumar wrote: > > > > On Fri, Jan 5, 2024 at 8:59 AM shveta malik wrote: > > > > > I was going the the patch set again, I have a question. The below > > commen

Re: Synchronizing slots from primary to standby

2024-01-05 Thread Dilip Kumar
ING, and ENABLED. The state transition process + * between these values is the same as the two_phase option (see TWO_PHASE + * TRANSACTIONS for details). -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Adding facility for injection points (or probe points?) for more advanced tests

2024-01-05 Thread Dilip Kumar
On Tue, Dec 12, 2023 at 4:15 PM Michael Paquier wrote: > > On Tue, Dec 12, 2023 at 10:27:09AM +0530, Dilip Kumar wrote: > > Oops, I only included the code changes where I am adding injection > > points and some comments to verify that, but missed the actual test > > file.

Re: Synchronizing slots from primary to standby

2024-01-03 Thread Dilip Kumar
then we can expand the behavior by providing an additional GUC with dbname. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: SLRU optimization - configurable buffer pool and partitioning the SLRU lock

2024-01-02 Thread Dilip Kumar
ant to rethink > the way the SLRU is extended -- page at a time wouldn't really make > sense, but preallocating an entire file might. Yeah, this is indeed an interesting idea. So I think if we are interested in working in this direction maybe this can be submitted in a different thread, IMHO. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: index prefetching

2023-12-20 Thread Dilip Kumar
+ */ + for (int i = 1; i < PREFETCH_SEQ_PATTERN_BLOCKS; i++) Correct, I think if we fetch this forward it will have an advantage with memory prefetching. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: SLRU optimization - configurable buffer pool and partitioning the SLRU lock

2023-12-18 Thread Dilip Kumar
nt counts. So as per my understanding by distributing the SLRU locks there are scenarios where we will not need group update anymore but OTOH there are also scenarios where we will still benefit from the group update. [1] https://www.postgresql.org/message-id/CAFiTN-u-XEzhd%3DhNGW586fmQwdT

Re: SLRU optimization - configurable buffer pool and partitioning the SLRU lock

2023-12-14 Thread Dilip Kumar
On Thu, Dec 14, 2023 at 4:36 PM Dilip Kumar wrote: > > On Wed, Dec 13, 2023 at 5:49 PM Andrey M. Borodin > wrote: > > > > On 12 Dec 2023, at 18:28, Alvaro Herrera wrote: > > > > > > Andrey, do you have any stress tests or anything else that you use

Re: SLRU optimization - configurable buffer pool and partitioning the SLRU lock

2023-12-14 Thread Dilip Kumar
4| 512/1024| 44 patch-2 | 1024| 512/1024| 72 [1] https://www.postgresql.org/message-id/flat/e46cdea96979545b2d8a13b451d8b1ce61dc7238.camel%40postgrespro.ru#0ed2cad11470825d464093fe6b8ef6a3 -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: logical decoding and replication of sequences, take 2

2023-12-13 Thread Dilip Kumar
On Thu, Dec 14, 2023 at 12:31 PM Ashutosh Bapat wrote: > > On Thu, Dec 14, 2023 at 10:53 AM Dilip Kumar wrote: > > > > > > > > > > > It is correct that we can make a wrong decision about whether a change > > > is transactional or no

Re: logical decoding and replication of sequences, take 2

2023-12-13 Thread Dilip Kumar
e sequence relids to the hash, I mean that hash is only maintained for deciding whether the sequence is changed in that transaction or not. So no adding such relids to hash seems like a root cause of the issue. Honestly, I haven't analyzed this idea in detail about how easy it would be to add on

Re: Improve eviction algorithm in ReorderBuffer

2023-12-12 Thread Dilip Kumar
lest transaction in this list are not very significant. And remaining in the small transaction list and from the small transaction list we can choose to spill multiple transactions at a time. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: SLRU optimization - configurable buffer pool and partitioning the SLRU lock

2023-12-12 Thread Dilip Kumar
d if that exception occurred we will release the old bank lock and acquire a new lock. This case might not be performant because now it is possible that after getting the lock leader might need to wait again on another bank lock, but this is an extremely exceptional case so should not be worried about

Re: Adding facility for injection points (or probe points?) for more advanced tests

2023-12-11 Thread Dilip Kumar
On Mon, Dec 11, 2023 at 3:14 PM Michael Paquier wrote: > > On Mon, Dec 11, 2023 at 11:09:45AM +0530, Dilip Kumar wrote: > > I haven't specifically done a review or testing of this patch, but I > > have used this for testing the CLOG group update code with my > > SLRU-spec

Re: Improve eviction algorithm in ReorderBuffer

2023-12-11 Thread Dilip Kumar
ome under memory limit instead of selecting a couple of large transactions which are consuming 8-9%? > > As a side note, the idea (c) mentioned in the comment, evicting > multiple transactions at once to free a given portion of the memory, > would also help in avoiding back and forth the memory threshold. It's > also worth considering. Yes, I think it is worth considering. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Synchronizing slots from primary to standby

2023-12-11 Thread Dilip Kumar
On Mon, Dec 11, 2023 at 2:21 PM shveta malik wrote: > > On Mon, Dec 11, 2023 at 1:47 PM Dilip Kumar wrote: > > > > On Fri, Dec 8, 2023 at 2:36 PM Amit Kapila wrote: > > > > > > On Wed, Dec 6, 2023 at 4:53 PM shveta malik > > > wrote: > > >

Re: Synchronizing slots from primary to standby

2023-12-11 Thread Dilip Kumar
sibility. I mean if the user has set 'standby_slot_names' on standby then let standby also wait for cascading standby to sync their slots? Is there any issue with that behavior? -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: trying again to get incremental backup

2023-12-10 Thread Dilip Kumar
On Mon, Dec 11, 2023 at 11:44 AM Dilip Kumar wrote: > > On Tue, Dec 5, 2023 at 11:40 PM Robert Haas wrote: > > > > On Mon, Dec 4, 2023 at 3:58 PM Robert Haas wrote: > > > Considering all this, what I'm inclined to do is go and put > > > UPLOAD_MANIFEST ba

Re: trying again to get incremental backup

2023-12-10 Thread Dilip Kumar
se backup: ERROR: manifest requires WAL from final timeline 1 ending at 0/6F8, but this backup starts at 0/628 pg_basebackup: removing data directory "d1" -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: Adding facility for injection points (or probe points?) for more advanced tests

2023-12-10 Thread Dilip Kumar
DzNoKU%2B8FP-S6zpv-r4Gm-Y%2BQ%40mail.gmail.com -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: logical decoding and replication of sequences, take 2

2023-12-06 Thread Dilip Kumar
tion in the same transaction, but that's not an issue because if some of the changes of this transaction are seen when snapbuild state < SNAPBUILD_FULL_SNAPSHOT then this transaction has to get committed before the state change to SNAPBUILD_CONSISTENT_SNAPSHOT i.e. the commit LSN of this transaction is

Re: logical decoding and replication of sequences, take 2

2023-12-06 Thread Dilip Kumar
On Wed, Dec 6, 2023 at 7:17 PM Tomas Vondra wrote: > > On 12/6/23 12:05, Dilip Kumar wrote: > > On Wed, Dec 6, 2023 at 3:36 PM Amit Kapila wrote: > >> > >>> Why can't we use the same concept of > >>> SnapBuildDistributeNewCatalogSnapshot(), I mea

Re: logical decoding and replication of sequences, take 2

2023-12-06 Thread Dilip Kumar
e changes in a separate queue in the order they arrive and as soon as we process the next commit we will process all the non-transactional changes at that time. Do you see issue with that? -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: logical decoding and replication of sequences, take 2

2023-12-06 Thread Dilip Kumar
On Wed, Dec 6, 2023 at 11:12 AM Dilip Kumar wrote: > > On Sun, Dec 3, 2023 at 11:22 PM Tomas Vondra > wrote: > > I was also wondering what happens if the sequence changes are transactional but somehow the snap builder state changes to SNAPBUILD_FULL_SNAPSHOT in bet

Re: logical decoding and replication of sequences, take 2

2023-12-05 Thread Dilip Kumar
tional sequence change so that while sending it to downstream whenever it is necessary we will change the historic snapshot? -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: SLRU optimization - configurable buffer pool and partitioning the SLRU lock

2023-11-29 Thread Dilip Kumar
async.h:15 > > error: src/include/commands/async.h: patch does not apply > > Yeah, this patch series conflicts with today's commit 4ed8f0913bfd. I will send a rebased version by tomorrow. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: SLRU optimization - configurable buffer pool and partitioning the SLRU lock

2023-11-23 Thread Dilip Kumar
On Thu, Nov 23, 2023 at 11:34 AM Dilip Kumar wrote: > > Note: With this testing, we have found a bug in the bank-wise > approach, basically we are clearing a procglobal->clogGroupFirst, even > before acquiring the bank lock that means in most of the cases there > will be a sing

Re: SLRU optimization - configurable buffer pool and partitioning the SLRU lock

2023-11-22 Thread Dilip Kumar
On Tue, Nov 21, 2023 at 2:03 PM Dilip Kumar wrote: > > On Mon, Nov 20, 2023 at 4:42 PM Dilip Kumar wrote: > > > > On Mon, Nov 20, 2023 at 2:37 PM Andrey M. Borodin > > wrote: > > > > > > On 20 Nov 2023, at 13:51, Dilip Kumar wrote: > > >

Re: SLRU optimization - configurable buffer pool and partitioning the SLRU lock

2023-11-20 Thread Dilip Kumar
On Mon, Nov 20, 2023 at 2:37 PM Andrey M. Borodin wrote: > > On 20 Nov 2023, at 13:51, Dilip Kumar wrote: > > > > 2) Do we really need one separate lwlock tranche for each SLRU? > > > > IMHO if we use the same lwlock tranche then the wait event will show >

Re: SLRU optimization - configurable buffer pool and partitioning the SLRU lock

2023-11-20 Thread Dilip Kumar
On Fri, Nov 17, 2023 at 7:28 PM Alvaro Herrera wrote: > > On 2023-Nov-17, Dilip Kumar wrote: I think I need some more clarification for some of the review comments > > On Thu, Nov 16, 2023 at 3:11 PM Alvaro Herrera > > wrote: > > > > > > I just

Re: SLRU optimization - configurable buffer pool and partitioning the SLRU lock

2023-11-19 Thread Dilip Kumar
ther external functions starting with "SimpleLruGet", are you fine with this name? -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: SLRU optimization - configurable buffer pool and partitioning the SLRU lock

2023-11-18 Thread Dilip Kumar
esitant to go > without any testing here. Maybe it'd be possible to use Michael > Paquier's injection points somehow? Sorry, but I am not aware of "Michael Paquier's injection" Is it something already in the repo? Can you redirect me to some of the example test cases if we alread

Re: SLRU optimization - configurable buffer pool and partitioning the SLRU lock

2023-11-16 Thread Dilip Kumar
ince useful values are below a couple thousand anyhow. I agree, that allowing xact_buffers to grow beyond 65536 up to the slru.h-defined limit of 131072 is not that bad, so I will change that in the next version. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: trying again to get incremental backup

2023-11-14 Thread Dilip Kumar
tra '*limit_block' out parameter which it is actually reading from the entry itself? -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: trying again to get incremental backup

2023-11-13 Thread Dilip Kumar
On Tue, Nov 14, 2023 at 12:52 AM Robert Haas wrote: > > On Fri, Nov 10, 2023 at 6:27 AM Dilip Kumar wrote: > > - I think 0001 looks good improvement irrespective of the patch series. > > OK, perhaps that can be independently committed, then, if nobody objects. > > Tha

Re: trying again to get incremental backup

2023-11-10 Thread Dilip Kumar
_access(), + errmsg("could not write file \"%s\": %m", + FilePathName(io->file; /could not write file/ could not read file 8. +/* + * Comparator to sort a List of WalSummaryFile objects by start_lsn. + */ +static int +ListComparatorForWalSummaryFiles(const ListCell *a, const ListCell *b) +{ -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: SLRU optimization - configurable buffer pool and partitioning the SLRU lock

2023-11-09 Thread Dilip Kumar
lone and only adjust the one with which they have a performance > problems; it's not going to be the same one for everybody. +1 -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: SLRU optimization - configurable buffer pool and partitioning the SLRU lock

2023-11-09 Thread Dilip Kumar
On Thu, Nov 9, 2023 at 9:39 PM Robert Haas wrote: > > On Wed, Nov 8, 2023 at 6:41 AM Dilip Kumar wrote: > > Here is the updated version of the patch, here I have taken the > > approach suggested by Andrey and I discussed the same with Alvaro > > offlist and he also agre

Re: SLRU optimization - configurable buffer pool and partitioning the SLRU lock

2023-11-07 Thread Dilip Kumar
ough I don't have any number which proves this. Currently, I want to focus on all the base patches and keep this patch as add on and later if we find its useful and want to pursue this then we will see how to make it better readable. > > Subject: [PATCH v4 5/5] Ensure slru buffer slots are in multiple of numbe of > partitions > > I think the 0005 patch can be merged to 0001. Yeah in the next version, it is done that way. Planning to post end of the day. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: SLRU optimization - configurable buffer pool and partitioning the SLRU lock

2023-11-06 Thread Dilip Kumar
On Mon, Nov 6, 2023 at 1:05 PM Andrey M. Borodin wrote: > > On 6 Nov 2023, at 09:09, Dilip Kumar wrote: > > > > > >> Having hashtable to find SLRU page in the buffer IMV is too slow. Some > >> comments on this approach can be found here [0]. > >> I'

Re: SLRU optimization - configurable buffer pool and partitioning the SLRU lock

2023-11-05 Thread Dilip Kumar
On Sun, Nov 5, 2023 at 1:37 AM Andrey M. Borodin wrote: > On 30 Oct 2023, at 09:20, Dilip Kumar wrote: > > changed the logic of SlruAdjustNSlots() in 0002, such that now it > starts with the next power of 2 value of the configured slots and > keeps doubling the number of banks

Re: Parallel Bitmap Heap Scan reports per-worker stats in EXPLAIN ANALYZE

2023-11-04 Thread Dilip Kumar
backend and the worker? I mean here for the backend you are showing lossy pages only if it is > 0 whereas for workers we are showing 0 lossy pages as well. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: SLRU optimization - configurable buffer pool and partitioning the SLRU lock

2023-10-25 Thread Dilip Kumar
On Wed, Oct 25, 2023 at 5:58 PM Amit Kapila wrote: > > On Fri, Oct 20, 2023 at 9:40 AM Dilip Kumar wrote: > > > > On Sat, Oct 14, 2023 at 9:43 AM Amit Kapila wrote: > > > > > > This and other results shared by you look promising. Will there be any > >

Re: SLRU optimization - configurable buffer pool and partitioning the SLRU lock

2023-10-24 Thread Dilip Kumar
On Tue, Oct 24, 2023 at 9:34 PM Alvaro Herrera wrote: > > On 2023-Oct-11, Dilip Kumar wrote: > > > In my last email, I forgot to give the link from where I have taken > > the base path for dividing the buffer pool in banks so giving the same > > here[1]. And loo

Re: SLRU optimization - configurable buffer pool and partitioning the SLRU lock

2023-10-19 Thread Dilip Kumar
under the same buffer pool and comparing against relation pages for victim buffer selection might cause different problems. But anyway I would discuss those points maybe in that thread. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: pg_upgrade's interaction with pg_resetwal seems confusing

2023-10-13 Thread Dilip Kumar
control file by itself with the required value of OID. The point I am > slightly worried about doing via server-side function is that some > online and or shutdown checkpoint can update other values in the > control file as well whereas if we do via pg_upgrade, we can have > bet

Re: pg_upgrade's interaction with pg_resetwal seems confusing

2023-10-12 Thread Dilip Kumar
t is what we want no? I mean instead of resetwal directly modifying the control file we will modify that value in the server using the binary_upgrade function and then have that value flush to the disk by shutdown checkpoint. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: SLRU optimization - configurable buffer pool and partitioning the SLRU lock

2023-10-12 Thread Dilip Kumar
On Wed, Oct 11, 2023 at 5:57 PM Dilip Kumar wrote: > > On Wed, Oct 11, 2023 at 4:34 PM Dilip Kumar wrote: > In my last email, I forgot to give the link from where I have taken > the base path for dividing the buffer pool in banks so giving the same > here[1]. And looking at this

Re: SLRU optimization - configurable buffer pool and partitioning the SLRU lock

2023-10-11 Thread Dilip Kumar
On Wed, Oct 11, 2023 at 4:34 PM Dilip Kumar wrote: > > The small size of the SLRU buffer pools can sometimes become a > performance problem because it’s not difficult to have a workload > where the number of buffers actively in use is larger than the > fixed-size buffer pool

SLRU optimization - configurable buffer pool and partitioning the SLRU lock

2023-10-11 Thread Dilip Kumar
actOffsetBuffer 687 | 427 LWLock | BufferContent Credits: - The base patch v1-0001 is authored by Thomas Munro and I have just rebased it. - 0002 and 0003 are new patches written by me based on design ideas from Robert and Myself. -- Regards, Dilip Kum

Re: Opportunistically pruning page before update

2023-10-05 Thread Dilip Kumar
other target pages as well which we are getting from FSM? Because in the patch you have put a check in a loop it will try to prune every page it gets from the FSM not just the current target page of the backend. Just wanted to understand if this is intentional. In general, all 4 ideas look

Re: [PoC] pg_upgrade: allow to upgrade publisher node

2023-10-05 Thread Dilip Kumar
lreceiver goes down, we won't be able to > send the required WAL and users won't be able to ensure that because > even after restart the same situation can happen. The ideal way is to > have something that puts the system in READ ONLY state during shutdown > and then we can probably al

Re: [PoC] pg_upgrade: allow to upgrade publisher node

2023-10-02 Thread Dilip Kumar
fter commit_flush_lsn no decodable record was generated then we are safe to upgrade that slot. So this seems an expandable approach. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

Re: [PoC] pg_upgrade: allow to upgrade publisher node

2023-09-25 Thread Dilip Kumar
On Mon, Sep 25, 2023 at 1:23 PM Bharath Rupireddy wrote: > > On Mon, Sep 25, 2023 at 12:32 PM Dilip Kumar wrote: > > > > > > Is there anything else that stops this patch from supporting migration > > > > of logical replication slots from PG versions < 17?

Re: [PoC] pg_upgrade: allow to upgrade publisher node

2023-09-25 Thread Dilip Kumar
On Mon, Sep 25, 2023 at 12:30 PM Dilip Kumar wrote: > > On Mon, Sep 25, 2023 at 11:15 AM Bharath Rupireddy > wrote: > > > > On Fri, Sep 22, 2023 at 12:11 PM Amit Kapila > > wrote: > > > > > > Yeah, both by tests and manually verifying the WAL re

Re: [PoC] pg_upgrade: allow to upgrade publisher node

2023-09-25 Thread Dilip Kumar
point and that is not yet synched to the disk then we are doing so. I think this is the most important change otherwise many slots for which we have already streamed all the WAL might give an error assuming that there are pending WAL from the slots which are not yet confirmed. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com

  1   2   3   4   5   6   7   8   9   10   >