孔凡深(云梳) wrote:
> Hi, Antonin. I am more interested in zheap. Recently reviewing the patch you
> submitted.
> When I use pg_undodump-tool to dump the undo page chunk, I found that some
> chunk header is abnormal.
> After reading the relevant codes in
> 0006-The-initial-implementation-of-the-pg
Hi,
On Thu, Nov 25, 2021 at 10:00 PM Antonin Houska wrote:
>
> So it should be ok if the temporary undo is managed and discarded by
> individual backends. Patch 0005 of the new series tries to do that.
The cfbot reports that at least the 001 patch doesn't apply anymore:
http://cfbot.cputube.org/
; > > >
> > > > >A transaction creates a temporary table, does some (many) changes
> > > > > and then
> > > > >gets rolled back. The undo records are being applied and it takes
> > > > > some
> > > > >
Thomas Munro wrote:
> On Wed, Sep 29, 2021 at 8:18 AM Antonin Houska wrote:
> > I'm just trying to use the existing infrastructure: the effect of DROP TABLE
> > also appear to be performed by the checkpointer. However I don't know why
> > the
> > unlinks need to be performed by the checkpointer
On Wed, Sep 29, 2021 at 8:18 AM Antonin Houska wrote:
> I'm just trying to use the existing infrastructure: the effect of DROP TABLE
> also appear to be performed by the checkpointer. However I don't know why the
> unlinks need to be performed by the checkpointer.
For DROP TABLE, we leave an empt
Dmitry Dolgov <9erthali...@gmail.com> wrote:
> > On Tue, Sep 21, 2021 at 10:07:55AM +0200, Dmitry Dolgov wrote:
> > On Tue, 21 Sep 2021 09:00 Antonin Houska, wrote:
> >
> > > Dmitry Dolgov <9erthali...@gmail.com> wrote:
> > >
> > > > Yep, makes sense, thanks. I have few more questions:
> > > >
>
> On Tue, Sep 21, 2021 at 10:07:55AM +0200, Dmitry Dolgov wrote:
> On Tue, 21 Sep 2021 09:00 Antonin Houska, wrote:
>
> > Dmitry Dolgov <9erthali...@gmail.com> wrote:
> >
> > > Yep, makes sense, thanks. I have few more questions:
> > >
> > > * The use case with orphaned files is working somewhat d
porary table, does some (many) changes
> > > > and then
> > > >gets rolled back. The undo records are being applied and it takes
> > > > some
> > > >time. Since XID of the transaction did not affect
> > > > oldestFul
On Mon, Sep 27, 2021 at 7:43 PM Antonin Houska wrote:
>
> Amit Kapila wrote:
>
> Although the postgres core probably does not raise FATAL errors too often (OOM
> conditions seem to be the typical cause), I'm still not enthusiastic about
> idea that the undo feature turns such errors into PANIC.
>
Amit Kapila wrote:
> On Fri, Sep 24, 2021 at 4:44 PM Antonin Houska wrote:
> >
> > Amit Kapila wrote:
> >
> > > On Mon, Sep 20, 2021 at 10:24 AM Antonin Houska wrote:
> > > >
> > > > Amit Kapila wrote:
> > > >
> > > > > On Fri, Sep 17, 2021 at 9:50 PM Dmitry Dolgov <9erthali...@gmail.com>
>
On Fri, Sep 24, 2021 at 4:44 PM Antonin Houska wrote:
>
> Amit Kapila wrote:
>
> > On Mon, Sep 20, 2021 at 10:24 AM Antonin Houska wrote:
> > >
> > > Amit Kapila wrote:
> > >
> > > > On Fri, Sep 17, 2021 at 9:50 PM Dmitry Dolgov <9erthali...@gmail.com>
> > > > wrote:
> > > > >
> > > > > > On T
Amit Kapila wrote:
> On Mon, Sep 20, 2021 at 10:24 AM Antonin Houska wrote:
> >
> > Amit Kapila wrote:
> >
> > > On Fri, Sep 17, 2021 at 9:50 PM Dmitry Dolgov <9erthali...@gmail.com>
> > > wrote:
> > > >
> > > > > On Tue, Sep 14, 2021 at 10:51:42AM +0200, Antonin Houska wrote:
> > > >
> > > >
n did not affect
> > > oldestFullXidHavingUndo,
> > >the XID can disappear from the CLOG due to truncation.
> > >
> >
> > By above do you mean to say that in zheap code, we don't consider XIDs
> > that operate on temp table/undo for oldestFullXidH
On Mon, Sep 20, 2021 at 10:24 AM Antonin Houska wrote:
>
> Amit Kapila wrote:
>
> > On Fri, Sep 17, 2021 at 9:50 PM Dmitry Dolgov <9erthali...@gmail.com> wrote:
> > >
> > > > On Tue, Sep 14, 2021 at 10:51:42AM +0200, Antonin Houska wrote:
> > >
> > > * What happened with the idea of abandoning di
On Tue, 21 Sep 2021 09:00 Antonin Houska, wrote:
> Dmitry Dolgov <9erthali...@gmail.com> wrote:
>
> > Yep, makes sense, thanks. I have few more questions:
> >
> > * The use case with orphaned files is working somewhat differently after
> > the rebase on the latest master, do you observe it as w
Dmitry Dolgov <9erthali...@gmail.com> wrote:
> > On Tue, Sep 14, 2021 at 10:51:42AM +0200, Antonin Houska wrote:
> > Antonin Houska wrote:
> >
> > > Dmitry Dolgov <9erthali...@gmail.com> wrote:
> >
> > > > * By throwing at the patchset `make installcheck` I'm getting from time
> > > > to time
>
ct
> > oldestFullXidHavingUndo,
> >the XID can disappear from the CLOG due to truncation.
> >
>
> By above do you mean to say that in zheap code, we don't consider XIDs
> that operate on temp table/undo for oldestFullXidHavingUndo?
I was referring to the co
Amit Kapila wrote:
> On Fri, Sep 17, 2021 at 9:50 PM Dmitry Dolgov <9erthali...@gmail.com> wrote:
> >
> > > On Tue, Sep 14, 2021 at 10:51:42AM +0200, Antonin Houska wrote:
> >
> > * What happened with the idea of abandoning discard worker for the sake
> > of simplicity? From what I see limiting
On Fri, Sep 17, 2021 at 9:50 PM Dmitry Dolgov <9erthali...@gmail.com> wrote:
>
> > On Tue, Sep 14, 2021 at 10:51:42AM +0200, Antonin Houska wrote:
>
> * What happened with the idea of abandoning discard worker for the sake
> of simplicity? From what I see limiting everything to foreground undo
>
> On Tue, Sep 14, 2021 at 10:51:42AM +0200, Antonin Houska wrote:
> Antonin Houska wrote:
>
> > Dmitry Dolgov <9erthali...@gmail.com> wrote:
>
> > > * By throwing at the patchset `make installcheck` I'm getting from time
> > > to time
> > > and error on the restart:
> > >
> > > TRAP: Failed
On Thu, Sep 9, 2021 at 8:33 PM Antonin Houska wrote:
>
> The cfbot complained that the patch series no longer applies, so I've rebased
> it and also tried to make sure that the other flags become green.
>
> One particular problem was that pg_upgrade complained that "live undo data"
> remains in th
> On Wed, Jun 30, 2021 at 07:41:16PM +0200, Antonin Houska wrote:
> Antonin Houska wrote:
>
> > tsunakawa.ta...@fujitsu.com wrote:
> >
> > > I'm crawling like a snail to read the patch set. Below are my first set
> > > of review comments, which are all minor.
> >
> > Thanks.
>
> I've added the
On Wed, Jun 30, 2021 at 11:10 PM Antonin Houska wrote:
>
> Antonin Houska wrote:
>
> > tsunakawa.ta...@fujitsu.com wrote:
> >
> > > I'm crawling like a snail to read the patch set. Below are my first set
> > > of review comments, which are all minor.
> >
> > Thanks.
>
> I've added the patch to
Antonin Houska wrote:
> tsunakawa.ta...@fujitsu.com wrote:
>
> > I'm crawling like a snail to read the patch set. Below are my first set of
> > review comments, which are all minor.
>
> Thanks.
I've added the patch to the upcoming CF [1], so it possibly gets more review
and makes some progr
tsunakawa.ta...@fujitsu.com wrote:
> I'm crawling like a snail to read the patch set. Below are my first set of
> review comments, which are all minor.
Thanks. I will check your comments when I'll be preparing the next version of
the patch.
--
Antonin Houska
Web: https://www.cybertec-postgre
aracters with pg_waldump? pg_waldump uses -r
to filter by rmgr. pg_undodump can output record contents by default like
pg_waldump. Considering pg_dump and pg_dumpall also output all data by
default, that seems how PostgreSQL commands behave.
(12)
+ startsegendseg
startseg and endseg ar
On Wed, Feb 3, 2021 at 2:45 PM tsunakawa.ta...@fujitsu.com
wrote:
>
> From: Antonin Houska
> > not really an "ordinary patch".
> >
> > [1]
> > https://www.postgresql.org/message-id/CA%2BhUKG%2BMpzRsZFE7ChhR
> > q-Br5VYYi6mafVQ73Af7ahioWo5o8w%40mail.gmail.com
>
> I'm a bit interested in zheap-rela
From: Antonin Houska
> not really an "ordinary patch".
>
> [1]
> https://www.postgresql.org/message-id/CA%2BhUKG%2BMpzRsZFE7ChhR
> q-Br5VYYi6mafVQ73Af7ahioWo5o8w%40mail.gmail.com
I'm a bit interested in zheap-related topics. I'm reading this discussion to
see what I can do. (But this thread i
Bruce Momjian wrote:
> On Fri, Jan 29, 2021 at 06:30:15PM +0100, Antonin Houska wrote:
> > Antonin Houska wrote:
> > > Well, on repeated run of the test I could also hit the first one. I could
> > > fix
> > > it and will post a new version of the patch (along with some other small
> > > changes
On Fri, Jan 29, 2021 at 06:30:15PM +0100, Antonin Houska wrote:
> Antonin Houska wrote:
> > Well, on repeated run of the test I could also hit the first one. I could
> > fix
> > it and will post a new version of the patch (along with some other small
> > changes) this week.
>
> Attached is the n
Dmitry Dolgov <9erthali...@gmail.com> wrote:
> Thanks for the updated patch. As I've mentioned off the list I'm slowly
> looking through it with the intent to concentrate on undo progress
> tracking. But before I will post anything I want to mention couple of
> strange issues I see, otherwise I wi
> On Fri, Dec 04, 2020 at 10:22:42AM +0100, Antonin Houska wrote:
> Amit Kapila wrote:
>
> > On Fri, Nov 13, 2020 at 6:02 PM Antonin Houska wrote:
> > >
> > > Amit Kapila wrote:
> > >
> > > > On Thu, Nov 12, 2020 at 2:45 PM Antonin Houska wrote:
> > > >
> > > > If you want to track at undo reco
On Fri, Dec 4, 2020 at 1:50 PM Antonin Houska wrote:
>
> Amit Kapila wrote:
>
>
> > The earlier version of the patch having all these ideas
> > implemented is attached
> > (Infrastructure-to-execute-pending-undo-actions and
> > Provide-interfaces-to-store-and-fetch-undo-records). The second one
>
Amit Kapila wrote:
> On Wed, Nov 25, 2020 at 8:00 PM Antonin Houska wrote:
> > I meant that AbortOutOfAnyTransaction should PANIC itself if it sees that
> > there is unapplied undo, so nothing changes for its callers. Do I still miss
> > something?
> >
>
> Adding PANIC in some generic code-pat
Amit Kapila wrote:
> On Wed, Nov 25, 2020 at 7:47 PM Antonin Houska wrote:
> >
> > Antonin Houska wrote:
> >
> > > Amit Kapila wrote:
> >
> > > > I think we also need to maintain oldestXidHavingUndo for CLOG
> > > > truncation and
> > > > transaction-wraparound. We can't allow CLOG truncation
On Wed, Nov 25, 2020 at 7:47 PM Antonin Houska wrote:
>
> Antonin Houska wrote:
>
> > Amit Kapila wrote:
>
> > > I think we also need to maintain oldestXidHavingUndo for CLOG truncation
> > > and
> > > transaction-wraparound. We can't allow CLOG truncation for the transaction
> > > whose undo i
On Wed, Nov 25, 2020 at 8:00 PM Antonin Houska wrote:
>
> Amit Kapila wrote:
>
> > On Wed, Nov 18, 2020 at 4:03 PM Antonin Houska wrote:
> > >
> > > Amit Kapila wrote:
> > >
> > > > On Fri, Nov 13, 2020 at 6:02 PM Antonin Houska wrote:
> > > > >
> > > > > Amit Kapila wrote:
> > > > >
> > > >
Amit Kapila wrote:
> On Wed, Nov 18, 2020 at 4:03 PM Antonin Houska wrote:
> >
> > Amit Kapila wrote:
> >
> > > On Fri, Nov 13, 2020 at 6:02 PM Antonin Houska wrote:
> > > >
> > > > Amit Kapila wrote:
> > > >
> > > > > On Thu, Nov 12, 2020 at 2:45 PM Antonin Houska
> > > > > wrote:
> > > >
Antonin Houska wrote:
> Amit Kapila wrote:
> > I think we also need to maintain oldestXidHavingUndo for CLOG truncation and
> > transaction-wraparound. We can't allow CLOG truncation for the transaction
> > whose undo is not discarded as that could be required by some other
> > transaction.
>
On Wed, Nov 18, 2020 at 4:03 PM Antonin Houska wrote:
>
> Amit Kapila wrote:
>
> > On Fri, Nov 13, 2020 at 6:02 PM Antonin Houska wrote:
> > >
> > > Amit Kapila wrote:
> > >
> > > > On Thu, Nov 12, 2020 at 2:45 PM Antonin Houska wrote:
> > > > >
> > > > >
> > > > > No background undo
> > > > >
Amit Kapila wrote:
> On Fri, Nov 13, 2020 at 6:02 PM Antonin Houska wrote:
> >
> > Amit Kapila wrote:
> >
> > > On Thu, Nov 12, 2020 at 2:45 PM Antonin Houska wrote:
> > > >
> > > >
> > > > No background undo
> > > > --
> > > >
> > > > Reduced complexity of the patch seems to b
On Sun, Nov 15, 2020 at 11:24 AM Amit Kapila wrote:
>
> On Fri, Nov 13, 2020 at 6:02 PM Antonin Houska wrote:
> >
> > Amit Kapila wrote:
> >
> > > On Thu, Nov 12, 2020 at 2:45 PM Antonin Houska wrote:
> > > >
> > > >
> > > > No background undo
> > > > --
> > > >
> > > > Reduced
On Fri, Nov 13, 2020 at 6:02 PM Antonin Houska wrote:
>
> Amit Kapila wrote:
>
> > On Thu, Nov 12, 2020 at 2:45 PM Antonin Houska wrote:
> > >
> > >
> > > No background undo
> > > --
> > >
> > > Reduced complexity of the patch seems to be the priority at the moment.
> > > Amit
>
Amit Kapila wrote:
> On Thu, Nov 12, 2020 at 2:45 PM Antonin Houska wrote:
> >
> >
> > No background undo
> > --
> >
> > Reduced complexity of the patch seems to be the priority at the moment. Amit
> > suggested that cleanup of an orphaned relation file is simple enough to be
> >
Thomas Munro wrote:
> On Thu, Nov 12, 2020 at 10:15 PM Antonin Houska wrote:
> I saw that -- great news! -- and have been meaning to write for a
> while. I think I am nearly ready to talk about it again.
I'm looking forward to it :-)
> 100% that it's worth trying to do something much simpler
On Thu, Nov 12, 2020 at 2:45 PM Antonin Houska wrote:
>
>
> No background undo
> --
>
> Reduced complexity of the patch seems to be the priority at the moment. Amit
> suggested that cleanup of an orphaned relation file is simple enough to be
> done on foreground and I agree.
>
Yea
On Thu, Nov 12, 2020 at 10:15 PM Antonin Houska wrote:
> Thomas Munro wrote:
> > Thanks. We decided to redesign a couple of aspects of the undo
> > storage and record layers that this patch was intended to demonstrate,
> > and work on that is underway. More on that soon.
>
> As my boss expresse
On Sat, Nov 30, 2019 at 9:25 PM Michael Paquier wrote:
> On Thu, Feb 07, 2019 at 07:35:31AM +0530, Andres Freund wrote:
> > It was JUST added ... :) thought I saw you reply on the other thread
> > about it, but I was wrong...
>
> Six months later without any activity, I am marking this entry as
>
On Thu, Feb 07, 2019 at 07:35:31AM +0530, Andres Freund wrote:
> It was JUST added ... :) thought I saw you reply on the other thread
> about it, but I was wrong...
Six months later without any activity, I am marking this entry as
returned with feedback. The latest patch set does not apply anymo
On Thu, Nov 28, 2019 at 3:45 PM Michael Paquier wrote:
> On Tue, Sep 17, 2019 at 10:03:20AM +1200, Thomas Munro wrote:
> > Oops, right. So it should just be added to the if condition. Will do.
>
> It's been a couple of months and the discussion has stale. It seems
> also that the patch was wait
On Tue, Sep 17, 2019 at 10:03:20AM +1200, Thomas Munro wrote:
> Oops, right. So it should just be added to the if condition. Will do.
It's been a couple of months and the discussion has stale. It seems
also that the patch was waiting for an update. So I am marking it as
RwF for now. Please fe
On Mon, Sep 16, 2019 at 10:37 PM Robert Haas wrote:
>
> It seems to me that zheap went wrong in ending up with separate undo
> types for in-place and non-in-place updates. Why not just have ONE
> kind of undo record that describes an update, and allow that update to
> have either one TID or two TI
On Tue, Sep 17, 2019 at 3:09 AM Kuntal Ghosh wrote:
> On Mon, Sep 16, 2019 at 11:23 AM Thomas Munro wrote:
> > Agreed. I added a line to break out of that loop if !block->in_use.
> >
> I think we should skip the block if !block->in_use. Because, the undo
> buffer can be registered in a subsequen
On Mon, Sep 16, 2019 at 11:09 AM Kuntal Ghosh
wrote:
> Okay. In that case, we need to rethink the cases for multi-inserts and
> non-inlace updates both of which currently inserts multiple undo
> record corresponding to a single WAL record. For multi-inserts, it can
> be solved easily by moving all
Hello Thomas,
On Mon, Sep 16, 2019 at 11:23 AM Thomas Munro wrote:
>
> > 1. In UndoLogAllocateInRecovery, when we find the current log number
> > from the list of registered blocks, we don't check whether the
> > block->in_use flag is true or not. In XLogResetInsertion, we just
> > reset in_use f
On Mon, Sep 16, 2019 at 5:27 AM Kuntal Ghosh wrote:
> While testing zheap over undo apis, we've found the following
> issues/scenarios that might need some fixes/discussions:
Thanks!
> 1. In UndoLogAllocateInRecovery, when we find the current log number
> from the list of registered blocks, we d
Hello Thomas,
While testing zheap over undo apis, we've found the following
issues/scenarios that might need some fixes/discussions:
1. In UndoLogAllocateInRecovery, when we find the current log number
from the list of registered blocks, we don't check whether the
block->in_use flag is true or no
On 2019-Sep-06, vignesh C wrote:
> Hi Thomas,
>
> While testing one of the recovery scenarios I found one issue:
> FailedAssertion("!(logno == context->recovery_logno)
I marked this patch Waiting on Author.
--
Álvaro Herrerahttps://www.2ndQuadrant.com/
PostgreSQL Development, 2
Hi Thomas,
While testing one of the recovery scenarios I found one issue:
FailedAssertion("!(logno == context->recovery_logno)
The details of the same is mentioned below:
The context's try_location was not updated in
UndoLogAllocateInRecovery, in PrepareUndoInsert the try_location was
updated wit
On Fri, Aug 30, 2019 at 8:27 PM Kuntal Ghosh wrote:
> I'm getting the following assert failure while performing the recovery
> with the same.
> "TRAP: FailedAssertion("slot->meta.status == UNDO_LOG_STATUS_FULL",
> File: "undolog.c", Line: 997)"
>
> I found that we don't emit an WAL record when we
Hello Thomas,
I was doing some testing for the scenario where the undo written by a
transaction overflows to multiple undo logs. For that I've modified
the following macro:
#define UndoLogMaxSize (1024 * 1024) /* 1MB undo log size */
(I should have used the provided pg_force_switch_undo t
On Wed, Aug 21, 2019 at 4:34 PM Robert Haas wrote:
> ReleaseResourcesAndProcessUndo() is only supposed to be called after
> AbortTransaction(), and the additional steps it performs --
> AtCleanup_Portals() and AtEOXact_Snapshot() or alternatively
> AtSubCleanup_Portals -- are taken from Cleanup(Su
o log. Suppose each undo log
has three states: (1) nobody's attached, (2) somebody's attached, and
(3) nobody's attached but the last record might need a fixup. When we
start up, all undo logs are in state 3, and the discard worker runs
around and puts them into state 1. Subsequen
On Wed, Aug 21, 2019 at 4:44 AM Andres Freund wrote:
> On 2019-08-20 21:02:18 +1200, Thomas Munro wrote:
> > 1. Anyone is allowed to try to read or write data at any UndoRecPtr
> > that has been allocated, through the buffer pool (though you'd usually
> > want to check it with UndoRecPtrIsDiscard
On Thu, Aug 22, 2019 at 9:55 PM Andres Freund wrote:
>
> Hi
>
> On August 22, 2019 9:14:10 AM PDT, Dilip Kumar wrote:
> > But, those requests will
> >ultimately be used for collecting the record by the bulk fetch. So if
> >we are planning to change the bulk fetch to read forward then maybe we
>
On Thu, Aug 22, 2019 at 7:34 PM Robert Haas wrote:
>
> On Thu, Aug 22, 2019 at 1:34 AM Dilip Kumar wrote:
> > Yeah, we can handle the bulk fetch as you suggested and it will make
> > it a lot easier. But, currently while registering the undo request
> > (especially during the first pass) we need
On Thu, Aug 22, 2019 at 12:54 AM Andres Freund wrote:
> But why? It makes a *lot* more sense to have it in the beginning. I
> don't think bulk-fetch really requires it to be in the end - we can
> still process records forward on a page-by-page basis.
There are two separate needs here: to be able
Hi
On August 22, 2019 9:14:10 AM PDT, Dilip Kumar wrote:
> But, those requests will
>ultimately be used for collecting the record by the bulk fetch. So if
>we are planning to change the bulk fetch to read forward then maybe we
>don't need the valid last undo record pointer because that we will
>
On Thu, Aug 22, 2019 at 9:21 PM Dilip Kumar wrote:
>
> On Thu, Aug 22, 2019 at 7:34 PM Robert Haas wrote:
> >
> > On Thu, Aug 22, 2019 at 1:34 AM Dilip Kumar wrote:
> > > Yeah, we can handle the bulk fetch as you suggested and it will make
> > > it a lot easier. But, currently while registering
On Thu, Aug 22, 2019 at 1:34 AM Dilip Kumar wrote:
> Yeah, we can handle the bulk fetch as you suggested and it will make
> it a lot easier. But, currently while registering the undo request
> (especially during the first pass) we need to compute the from_urecptr
> and the to_urecptr. And, for c
On Thu, Aug 22, 2019 at 11:04 AM Dilip Kumar wrote:
>
> On Thu, Aug 22, 2019 at 10:24 AM Andres Freund wrote:
> >
> > Hi,
> >
> > On 2019-08-22 10:19:04 +0530, Dilip Kumar wrote:
> > > On Thu, Aug 22, 2019 at 9:58 AM Andres Freund wrote:
> > > >
> > > > Hi,
> > > >
> > > > On 2019-08-22 09:51:22
On Thu, Aug 22, 2019 at 10:24 AM Andres Freund wrote:
>
> Hi,
>
> On 2019-08-22 10:19:04 +0530, Dilip Kumar wrote:
> > On Thu, Aug 22, 2019 at 9:58 AM Andres Freund wrote:
> > >
> > > Hi,
> > >
> > > On 2019-08-22 09:51:22 +0530, Dilip Kumar wrote:
> > > > We can not know the complete size of the
On Thu, Aug 22, 2019 at 9:58 AM Andres Freund wrote:
>
> Hi,
>
> On 2019-08-22 09:51:22 +0530, Dilip Kumar wrote:
> > We can not know the complete size of the record even by reading the
> > header because we have a payload that is variable part and payload
> > length are stored in the payload head
Hi,
On 2019-08-22 10:19:04 +0530, Dilip Kumar wrote:
> On Thu, Aug 22, 2019 at 9:58 AM Andres Freund wrote:
> >
> > Hi,
> >
> > On 2019-08-22 09:51:22 +0530, Dilip Kumar wrote:
> > > We can not know the complete size of the record even by reading the
> > > header because we have a payload that is
Hi,
On 2019-08-22 09:51:22 +0530, Dilip Kumar wrote:
> We can not know the complete size of the record even by reading the
> header because we have a payload that is variable part and payload
> length are stored in the payload header which again can be at random
> offset.
Wait, but that's just pu
On Wed, Aug 21, 2019 at 9:04 PM Robert Haas wrote:
>
> On Wed, Aug 21, 2019 at 3:55 AM Dilip Kumar wrote:
> > I have already attempted that part and I feel it is not making code
> > any simpler than what we have today. For packing, it's fine because I
> > can process all the member once and dire
On Wed, Aug 14, 2019 at 2:57 AM Andres Freund wrote:
> - My reading of the current xact.c integration is that it's not workable
> as is. Undo is executed outside of a valid transaction state,
> exceptions aren't properly undone, logic would need to be duplicated
> to a significant degree, ne
On August 21, 2019 8:36:34 AM PDT, Robert Haas wrote:
> We treat LWLockAcquire() as a no-fail operation in many
>places; in my opinion, that elog(ERROR) that we have for too many
>LWLocks should be changed to elog(PANIC) precisely because we do treat
>LWLockAcquire() as no-fail in lots of place
On Wed, Aug 21, 2019 at 3:55 AM Dilip Kumar wrote:
> I have already attempted that part and I feel it is not making code
> any simpler than what we have today. For packing, it's fine because I
> can process all the member once and directly pack it into one memory
> chunk and I can insert that to
On Wed, Aug 21, 2019 at 6:38 AM Amit Kapila wrote:
> > FWIW, although I also thought of doing what you are describing here, I
> > think Andres's proposal is probably preferable, because it's simpler.
> > There's not really any reason why we can't take the buffer locks from
> > within the critical
On Tue, Aug 20, 2019 at 8:10 PM Robert Haas wrote:
>
> On Tue, Aug 20, 2019 at 2:42 AM Amit Kapila wrote:
> > > Well, my main point, which so far has largely been ignored, was that we
> > > may not acquire page locks when we still need to search for victim
> > > buffers later. If we don't need to
On Tue, Aug 20, 2019 at 7:57 PM Robert Haas wrote:
>
> On Mon, Aug 19, 2019 at 2:04 AM Dilip Kumar wrote:
> > Currently, In UnpackedUndoRecord we store all members directly which
> > are set by the caller. We store pointers to some header which are
> > allocated internally by the undo layer and
On Tue, Aug 13, 2019 at 8:11 AM Robert Haas wrote:
> > We can probably check the fxid queue and error queue to get that
> > value. However, I am not sure if that is sufficient because incase we
> > perform the request in the foreground, it won't be present in queues.
>
> Oh, I forgot about that r
On Mon, Aug 19, 2019 at 2:04 AM Dilip Kumar wrote:
> Currently, In UnpackedUndoRecord we store all members directly which
> are set by the caller. We store pointers to some header which are
> allocated internally by the undo layer and the caller need not worry
> about setting them. So now you ar
On Tue, Aug 20, 2019 at 5:02 AM Thomas Munro wrote:
> 3. UndoLogDiscard() uses DiscardBuffer() to invalidate any currently
> unpinned buffers, and marks as BM_DISCARDED any that happen to be
> pinned right now, so they can't be immediately invalidated. Such
> buffers are never written back and a
On 2019-08-20 09:44:23 -0700, Andres Freund wrote:
> On 2019-08-20 21:02:18 +1200, Thomas Munro wrote:
> > Aside from code changes based on review (and I have more to come of
> > those), the attached experimental patchset (also at
> > https://github.com/EnterpriseDB/zheap/tree/undo) has a new proto
quire (find or create) enough clean to
> >hold the maximum amount of undo needed. These buffers would be marked
> >as !BM_TAG_VALID | BUF_REFCOUNT_ONE.
> >
> > I assume that we'd make a) cheap by keeping it pinned for undo logs that
> > a backend is
Hi,
On 2019-08-20 21:02:18 +1200, Thomas Munro wrote:
> Aside from code changes based on review (and I have more to come of
> those), the attached experimental patchset (also at
> https://github.com/EnterpriseDB/zheap/tree/undo) has a new protocol
> that, I hope, allows for better concurrency, rel
Hi,
On 2019-08-20 09:08:29 -0400, Robert Haas wrote:
> On Sat, Aug 17, 2019 at 1:28 PM Andres Freund wrote:
> > The primary one in the context here is that if we do *not* have to lock
> > the buffers all ahead of time, we can simplify the interface. We
> > certainly can't lock the buffers over IO
On Mon, Aug 19, 2019 at 8:22 AM Amit Kapila wrote:
> One point to remember in this regard is that we do need to modify the
> LSN in undo pages after writing WAL, so all the undo pages need to be
> locked by that time or we again need to take the lock on them.
Uh, but a big part of the point of se
On Tue, Aug 20, 2019 at 2:42 AM Amit Kapila wrote:
> > Well, my main point, which so far has largely been ignored, was that we
> > may not acquire page locks when we still need to search for victim
> > buffers later. If we don't need to lock the pages up-front, but only do
> > so once we're actual
On Mon, Aug 19, 2019 at 5:16 PM Andres Freund wrote:
> Well, my main point, which so far has largely been ignored, was that we
> may not acquire page locks when we still need to search for victim
> buffers later. If we don't need to lock the pages up-front, but only do
> so once we're actually cop
On Sat, Aug 17, 2019 at 1:28 PM Andres Freund wrote:
> The primary one in the context here is that if we do *not* have to lock
> the buffers all ahead of time, we can simplify the interface. We
> certainly can't lock the buffers over IO (due to buffer reclaim) as
> we're doing right now, so we'd n
o
> b) if needed pin the page of older undo that we need to update (e.g. to
>update the next pointer)
> c) perform clock sweep etc to acquire (find or create) enough clean to
>hold the maximum amount of undo needed. These buffers would be marked
>as !BM_TAG_VALID | BUF_REF
On Tue, Aug 20, 2019 at 2:46 AM Andres Freund wrote:
>
> On 2019-08-19 17:52:24 +0530, Amit Kapila wrote:
> > On Sat, Aug 17, 2019 at 10:58 PM Andres Freund wrote:
> > >
> > > > Well, I don't understand why you're on about this. We've discussed it
> > > > a number of times but I'm still confused
On 2019-08-19 17:52:24 +0530, Amit Kapila wrote:
> On Sat, Aug 17, 2019 at 10:58 PM Andres Freund wrote:
> >
> > Hi,
> >
> > On 2019-08-17 12:05:21 -0400, Robert Haas wrote:
> > > On Wed, Aug 14, 2019 at 12:39 PM Andres Freund wrote:
> > > > > > Again, I think it's not ok to just assume you can l
On Sat, Aug 17, 2019 at 10:58 PM Andres Freund wrote:
>
> Hi,
>
> On 2019-08-17 12:05:21 -0400, Robert Haas wrote:
> > On Wed, Aug 14, 2019 at 12:39 PM Andres Freund wrote:
> > > > > Again, I think it's not ok to just assume you can lock an essentially
> > > > > unbounded number of buffers. This
On Wed, Aug 14, 2019 at 10:35 PM Andres Freund wrote:
>
>
> > > - When reading an undo record, the whole stage of UnpackUndoData()
> > > reading data into a the UndoPackContext is omitted, reading directly
> > > into the UnpackedUndoRecord. That removes one further copy of the
> > >
On Sat, Aug 17, 2019 at 9:35 PM Robert Haas wrote:
>
> On Wed, Aug 14, 2019 at 12:39 PM Andres Freund wrote:
> > > > Again, I think it's not ok to just assume you can lock an essentially
> > > > unbounded number of buffers. This seems almost guaranteed to result in
> > > > deadlocks. And there's
WAL logging rules" is sufficient
explanation, because all the locking here happens one or two layers away
from the WAL logging site - so it's absolutely *NOT* obvious that that's
the explanation. And I don't think any of the locking sites actually has
comments explaining wh
1 - 100 of 347 matches
Mail list logo
