Hello hackers,
Based on ideas from earlier discussions[1][2], here is an experimental
WIP patch to improve recovery speed by prefetching blocks. If you set
wal_prefetch_distance to a positive distance, measured in bytes, then
the recovery loop will look ahead in the WAL and call PrefetchBuffer()
Hi,
On 2021-02-12 00:42:04 +0100, Tomas Vondra wrote:
> Yeah, that's a good point. I think it'd make sense to keep track of recent
> FPIs and skip prefetching such blocks. But how exactly should we implement
> that, how many blocks do we need to track? If you get an FPI, how long
> should we skip
On 2/12/21 5:46 AM, Andres Freund wrote:
Hi,
On 2021-02-12 00:42:04 +0100, Tomas Vondra wrote:
Yeah, that's a good point. I think it'd make sense to keep track of recent
FPIs and skip prefetching such blocks. But how exactly should we implement
that, how many blocks do we need to track? If y
Greetings,
* Andres Freund (and...@anarazel.de) wrote:
> On 2021-02-12 00:42:04 +0100, Tomas Vondra wrote:
> > Yeah, that's a good point. I think it'd make sense to keep track of recent
> > FPIs and skip prefetching such blocks. But how exactly should we implement
> > that, how many blocks do we n
On 2/13/21 10:39 PM, Stephen Frost wrote:
Greetings,
* Andres Freund (and...@anarazel.de) wrote:
On 2021-02-12 00:42:04 +0100, Tomas Vondra wrote:
Yeah, that's a good point. I think it'd make sense to keep track of recent
FPIs and skip prefetching such blocks. But how exactly should we impl
Greetings,
* Tomas Vondra (tomas.von...@enterprisedb.com) wrote:
> On 2/13/21 10:39 PM, Stephen Frost wrote:
> >* Andres Freund (and...@anarazel.de) wrote:
> >>On 2021-02-12 00:42:04 +0100, Tomas Vondra wrote:
> >>>Yeah, that's a good point. I think it'd make sense to keep track of recent
> >>>FPI
On 2/15/21 12:18 AM, Stephen Frost wrote:
> Greetings,
>
> ...
>
I think there's potential for some significant optimization going
forward, but I think it's basically optimization over what we're doing
today. As this is already a nontrivial patch, I'd argue for doing so
separat
Greetings,
* Tomas Vondra (tomas.von...@enterprisedb.com) wrote:
> Right, I was just going to point out the FPIs are not necessary - what
> matters is the presence of long streaks of WAL records touching the same
> set of blocks. But people with workloads where this is common likely
> don't need t
Hi,
On 3/17/21 10:43 PM, Stephen Frost wrote:
> Greetings,
>
> * Tomas Vondra (tomas.von...@enterprisedb.com) wrote:
>> Right, I was just going to point out the FPIs are not necessary - what
>> matters is the presence of long streaks of WAL records touching the same
>> set of blocks. But people w
On Wed, Feb 12, 2020 at 7:52 PM Thomas Munro wrote:
> 1. It now uses effective_io_concurrency to control how many
> concurrent prefetches to allow. It's possible that we should have a
> different GUC to control "maintenance" users of concurrency I/O as
> discussed elsewhere[1], but I'm staying o
I tried my luck at a quick read of this patchset.
I didn't manage to go over 0005 though, but I agree with Tomas that
having this be configurable in terms of bytes of WAL is not very
user-friendly.
First of all, let me join the crowd chanting that this is badly needed;
I don't need to repeat what
Hi Alvaro,
On Sat, Mar 14, 2020 at 10:15 AM Alvaro Herrera
wrote:
> I tried my luck at a quick read of this patchset.
Thanks! Here's a new patch set, and some inline responses to your feedback:
> I didn't manage to go over 0005 though, but I agree with Tomas that
> having this be configurable
On 2020-Mar-17, Thomas Munro wrote:
Hi Thomas
> On Sat, Mar 14, 2020 at 10:15 AM Alvaro Herrera
> wrote:
> > I didn't manage to go over 0005 though, but I agree with Tomas that
> > having this be configurable in terms of bytes of WAL is not very
> > user-friendly.
>
> The primary control is no
On Wed, Mar 18, 2020 at 2:47 PM Alvaro Herrera wrote:
> On 2020-Mar-17, Thomas Munro wrote:
> > I didn't want counters that wrap at ~4 billion, but I did want to be
> > able to read and write concurrently without tearing. Instructions
> > like "lock xadd" would provide more guarantees that I don'
On 3/18/21 1:54 AM, Thomas Munro wrote:
> On Thu, Mar 18, 2021 at 12:00 PM Tomas Vondra
> wrote:
>> On 3/17/21 10:43 PM, Stephen Frost wrote:
>>> Guess I'm just not a fan of pushing out a change that will impact
>>> everyone by default, in a possibly negative way (or positive, though
>>> that does
On Fri, Mar 19, 2021 at 2:29 PM Tomas Vondra
wrote:
> On 3/18/21 1:54 AM, Thomas Munro wrote:
> > I'm now looking at Horiguchi-san and Heikki's patch[2] to remove
> > XLogReader's callbacks, to try to understand how these two patch sets
> > are related. I don't really like the way those callbacks
On 4/7/21 1:24 PM, Thomas Munro wrote:
> Here's rebase, on top of Horiguchi-san's v19 patch set. My patches
> start at 0007. Previously, there was a "nowait" flag that was passed
> into all the callbacks so that XLogReader could wait for new WAL in
> some cases but not others. This new version u
On Thu, Apr 8, 2021 at 3:27 AM Tomas Vondra
wrote:
> On 4/7/21 1:24 PM, Thomas Munro wrote:
> > I made one other simplifying change: previously, the prefetch module
> > would read the WAL up to the "written" LSN (so, allowing itself to
> > read data that had been written but not yet flushed to dis
On 4/8/21 1:46 PM, Thomas Munro wrote:
> On Thu, Apr 8, 2021 at 3:27 AM Tomas Vondra
> wrote:
>> On 4/7/21 1:24 PM, Thomas Munro wrote:
>>> I made one other simplifying change: previously, the prefetch module
>>> would read the WAL up to the "written" LSN (so, allowing itself to
>>> read data t
Here's some little language fixes.
BTW, before beginning "recovery", PG syncs all the data dirs.
This can be slow, and it seems like the slowness is frequently due to file
metadata. For example, that's an obvious consequence of an OS crash, after
which the page cache is empty. I've made a habit
On Fri, Apr 9, 2021 at 3:37 PM Justin Pryzby wrote:
> Here's some little language fixes.
Thanks! Done. I rewrote the gibberish comment that made you say
"XXX: what?". Pushed.
> BTW, before beginning "recovery", PG syncs all the data dirs.
> This can be slow, and it seems like the slowness is
On Sat, Apr 10, 2021 at 08:27:42AM +1200, Thomas Munro wrote:
> On Fri, Apr 9, 2021 at 3:37 PM Justin Pryzby wrote:
> > Here's some little language fixes.
>
> Thanks! Done. I rewrote the gibberish comment that made you say
> "XXX: what?". Pushed.
>
> > BTW, before beginning "recovery", PG syn
On Sat, Apr 10, 2021 at 8:37 AM Justin Pryzby wrote:
> Did you see this?
> https://www.postgresql.org/message-id/GV0P278MB0483490FEAC879DCA5ED583DD2739%40GV0P278MB0483.CHEP278.PROD.OUTLOOK.COM
>
> I meant to mail you so you could include it in the same commit, but forgot
> until now.
Done, thanks
hackers
Subject: Re: WIP: WAL prefetch (another approach)
On Sat, Apr 10, 2021 at 8:37 AM Justin Pryzby wrote:
> Did you see this?
> INVALID URI REMOVED
> 278MB0483490FEAC879DCA5ED583DD2739*40GV0P278MB0483.CHEP278.PROD.OUTLOO
> K.COM__;JQ!!NpxR!wcPrhiB2CaHRtywGoh9Ap0M-k
On Sat, Apr 10, 2021 at 2:16 AM Thomas Munro wrote:
>
In commit 1d257577e08d3e598011d6850fd1025858de8c8c, there is a change
in file format for stats, won't it require bumping
PGSTAT_FILE_FORMAT_ID?
Actually, I came across this while working on my today's commit
f5fc2f5b23 where I forgot to bump
On 11/13/20 3:20 AM, Thomas Munro wrote:
>
> ...
>
> I'm not really sure what to do about achive restore scripts that
> block. That seems to be fundamentally incompatible with what I'm
> doing here.
>
IMHO we can't do much about that, except for documenting it - if the
prefetch can't work becau
Greetings,
* Tomas Vondra (tomas.von...@enterprisedb.com) wrote:
> On 11/13/20 3:20 AM, Thomas Munro wrote:
> > I'm not really sure what to do about achive restore scripts that
> > block. That seems to be fundamentally incompatible with what I'm
> > doing here.
>
> IMHO we can't do much about th
Greetings,
* Thomas Munro (thomas.mu...@gmail.com) wrote:
> On Sat, Nov 14, 2020 at 4:13 AM Stephen Frost wrote:
> > * Tomas Vondra (tomas.von...@enterprisedb.com) wrote:
> > > On 11/13/20 3:20 AM, Thomas Munro wrote:
> > > > I'm not really sure what to do about achive restore scripts that
> > >
On Thu, Nov 19, 2020 at 10:00 AM Stephen Frost wrote:
> * Thomas Munro (thomas.mu...@gmail.com) wrote:
> > Hmm. Every time I try to think of a protocol change for the
> > restore_command API that would be acceptable, I go around the same
> > circle of thoughts about event flow and realise that wh
> On 10 May 2021, at 06:11, Thomas Munro wrote:
> On Thu, Apr 22, 2021 at 11:22 AM Stephen Frost wrote:
>> I tend to agree with the idea to revert it, perhaps a +0 on that, but if
>> others argue it should be fixed in-place, I wouldn’t complain about it.
>
> Reverted.
>
> Note: eelpout may re
Hi,
It's great you posted a new version of this patch, so I took a look a
brief look at it. The code seems in pretty good shape, I haven't found
any real issues - just two minor comments:
This seems a bit strange:
#define DEFAULT_DECODE_BUFFER_SIZE 0x1
Why not to define this as a simple
On Fri, Nov 26, 2021 at 11:32 AM Tomas Vondra
wrote:
> The results are pretty good / similar to previous results. Replaying the
> 1h worth of work on a smaller machine takes ~5:30h without prefetching
> (master or with prefetching disabled). With prefetching enabled this
> drops to ~2h (default co
On 11/26/21 22:16, Thomas Munro wrote:
On Fri, Nov 26, 2021 at 11:32 AM Tomas Vondra
wrote:
The results are pretty good / similar to previous results. Replaying the
1h worth of work on a smaller machine takes ~5:30h without prefetching
(master or with prefetching disabled). With prefetching ena
On Sat, Nov 27, 2021 at 12:34 PM Tomas Vondra
wrote:
> One thing that's not clear to me is what happened to the reasons why
> this feature was reverted in the PG14 cycle?
Reasons for reverting:
1. A bug in commit 323cbe7c, "Remove read_page callback from
XLogReader.". I couldn't easily revert
Thomas Munro writes:
> On Sat, Nov 27, 2021 at 12:34 PM Tomas Vondra
> wrote:
>> One thing that's not clear to me is what happened to the reasons why
>> this feature was reverted in the PG14 cycle?
> 3. A wild goose chase for bugs on Tom Lane's antique 32 bit PPC
> machine. Tom eventually repr
Hi Thomas,
I am unable to apply these new set of patches on HEAD. Can you please share
the rebased patch or if you have any work branch can you please point it
out, I will refer to it for the changes.
--
With Regards,
Ashutosh sharma.
On Tue, Nov 23, 2021 at 3:44 PM Thomas Munro wrote:
> On Mo
On Fri, Nov 26, 2021 at 9:47 PM Tom Lane wrote:
> Yeah ... on the one hand, that machine has shown signs of
> hard-to-reproduce flakiness, so it's easy to write off the failures
> I saw as hardware issues. On the other hand, the flakiness I've
> seen has otherwise manifested as kernel crashes, wh
On Fri, 26 Nov 2021 at 21:47, Tom Lane wrote:
>
> Yeah ... on the one hand, that machine has shown signs of
> hard-to-reproduce flakiness, so it's easy to write off the failures
> I saw as hardware issues. On the other hand, the flakiness I've
> seen has otherwise manifested as kernel crashes, wh
The actual hardware of this machine is a Mac Mini Core 2 Duo. I'm not
really clear how the emulation is done and whether it makes a
reasonable test environment or not.
Hardware Overview:
Model Name: Mac mini
Model Identifier: Macmini2,1
Processor Name: Intel Core 2 Duo
Greg Stark writes:
> But if you're interested and can explain the tests to run I can try to
> get the tests running on this machine:
I'm not sure that machine is close enough to prove much, but by all
means give it a go if you wish. My test setup was explained in [1]:
>> To recap, the test lash
What tools and tool versions are you using to build? Is it just GCC for PPC?
There aren't any special build processes to make a fat binary involved?
On Thu, 16 Dec 2021 at 23:11, Tom Lane wrote:
>
> Greg Stark writes:
> > But if you're interested and can explain the tests to run I can try to
>
Greg Stark writes:
> What tools and tool versions are you using to build? Is it just GCC for PPC?
> There aren't any special build processes to make a fat binary involved?
Nope, just "configure; make" using that macOS version's regular gcc.
regards, tom lane
I have
IBUILD:postgresql gsstark$ ls /usr/bin/*gcc*
/usr/bin/gcc
/usr/bin/gcc-4.0
/usr/bin/gcc-4.2
/usr/bin/i686-apple-darwin9-gcc-4.0.1
/usr/bin/i686-apple-darwin9-gcc-4.2.1
/usr/bin/powerpc-apple-darwin9-gcc-4.0.1
/usr/bin/powerpc-apple-darwin9-gcc-4.2.1
I'm guessing I should do CC=/usr/bin/pow
Greg Stark writes:
> I'm guessing I should do CC=/usr/bin/powerpc-apple-darwin9-gcc-4.2.1
> or maybe 4.0.1. What version is on your G4?
$ gcc -v
Using built-in specs.
Target: powerpc-apple-darwin9
Configured with: /var/tmp/gcc/gcc-5493~1/src/configure --disable-checking
-enable-werror --prefix=/
Hm. I seem to have picked a bad checkout. I took the last one before
the revert (45aa88fe1d4028ea50ba7d26d390223b6ef78acc). Or there's some
incompatibility with the emulation and the IPC stuff parallel workers
use.
2021-12-17 17:51:51.688 EST [50955] LOG: background worker "parallel
worker" (PID
On 12/17/21 23:56, Greg Stark wrote:
Hm. I seem to have picked a bad checkout. I took the last one before
the revert (45aa88fe1d4028ea50ba7d26d390223b6ef78acc). Or there's some
incompatibility with the emulation and the IPC stuff parallel workers
use.
2021-12-17 17:51:51.688 EST [50955] LOG: b
Greg Stark writes:
> Hm. I seem to have picked a bad checkout. I took the last one before
> the revert (45aa88fe1d4028ea50ba7d26d390223b6ef78acc).
FWIW, I think that's the first one *after* the revert.
> 2021-12-17 17:51:51.688 EST [50955] LOG: background worker "parallel
> worker" (PID 54073)
On Fri, 17 Dec 2021 at 18:40, Tom Lane wrote:
>
> Greg Stark writes:
> > Hm. I seem to have picked a bad checkout. I took the last one before
> > the revert (45aa88fe1d4028ea50ba7d26d390223b6ef78acc).
>
> FWIW, I think that's the first one *after* the revert.
Doh
But the bigger question is. Are
Greg Stark writes:
> But the bigger question is. Are we really concerned about this flaky
> problem? Is it worth investing time and money on? I can get money to
> go buy a G4 or G5 and spend some time on it. It just seems a bit...
> niche. But if it's a real bug that represents something broken on
Thomas Munro writes:
>> FWIW I don't think we include updates to typedefs.list in patches.
> Seems pretty harmless? And useful to keep around in development
> branches because I like to pgindent stuff...
As far as that goes, my habit is to pull down
https://buildfarm.postgresql.org/cgi-bin/typed
Hi,
On 2021-12-29 17:29:52 +1300, Thomas Munro wrote:
> > FWIW I don't think we include updates to typedefs.list in patches.
>
> Seems pretty harmless? And useful to keep around in development
> branches because I like to pgindent stuff...
I think it's even helpful. As long as it's done with a b
On Sat, Dec 5, 2020 at 7:27 AM Stephen Frost wrote:
> * Thomas Munro (thomas.mu...@gmail.com) wrote:
> > I just noticed this thread proposing to retire pg_standby on that
> > basis:
> >
> > https://www.postgresql.org/message-id/flat/20201029024412.GP5380%40telsasoft.com
> >
> > I'd be happy to see
Hi,
I did a bunch of tests on v15, mostly to asses how much could the
prefetching help. The most interesting test I did was this:
1) primary instance on a box with 16/32 cores, 64GB RAM, NVMe SSD
2) replica on small box with 4 cores, 8GB RAM, SSD RAID
3) pause replication on the replica (pg_wal
On Thu, Feb 4, 2021 at 1:40 PM Tomas Vondra
wrote:
> With master, it'd take ~16000 seconds to catch up. I don't have the
> exact number, because I got tired of waiting, but the estimate is likely
> accurate (judging by other tests and how regular the progress is).
>
> With WAL prefetching enabled
Greetings,
* Thomas Munro (thomas.mu...@gmail.com) wrote:
> Rebase attached.
> Subject: [PATCH v15 4/6] Prefetch referenced blocks during recovery.
> diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
> index 4b60382778..ac27392053 100644
> --- a/doc/src/sgml/config.sgml
> +++ b/doc
On 2/10/21 10:50 PM, Stephen Frost wrote:
>
...
>
+/*
+ * Scan the current record for block references, and consider prefetching.
+ *
+ * Return true if we processed the current record to completion and still have
+ * queue space to process a new record, and false if we saturated the I/O
+ * qu
Thomas Munro writes:
> Yeah, it would have been nice to include that but it'll have to be for
> v15 due to lack of time to convince myself that it was correct. I do
> intend to look into more concurrency of that kind for v15. I have
> pushed these patches, updated to be disabled by default.
I h
On 4/21/21 6:30 PM, Tom Lane wrote:
> Thomas Munro writes:
>> Yeah, it would have been nice to include that but it'll have to be for
>> v15 due to lack of time to convince myself that it was correct. I do
>> intend to look into more concurrency of that kind for v15. I have
>> pushed these patche
On Thu, Apr 22, 2021 at 8:07 AM Tomas Vondra
wrote:
> On 4/21/21 6:30 PM, Tom Lane wrote:
> > Thomas Munro writes:
> >> Yeah, it would have been nice to include that but it'll have to be for
> >> v15 due to lack of time to convince myself that it was correct. I do
> >> intend to look into more c
On Thu, Apr 22, 2021 at 8:16 AM Thomas Munro wrote:
> That wasn't my plan, but I admit that the timing was non-ideal. In
> any case, I'll dig into these failures and then consider options.
> More soon.
Yeah, this clearly needs more work. xlogreader.c is difficult to work
with and I think we nee
Greetings,
On Wed, Apr 21, 2021 at 19:17 Thomas Munro wrote:
> On Thu, Apr 22, 2021 at 8:16 AM Thomas Munro
> wrote:
> > That wasn't my plan, but I admit that the timing was non-ideal. In
> > any case, I'll dig into these failures and then consider options.
> > More soon.
>
> Yeah, this clearl
Stephen Frost writes:
> On Wed, Apr 21, 2021 at 19:17 Thomas Munro wrote:
>> ... Personally I think the right thing to do now is to revert it
>> and re-propose for 15 early in the cycle, supported with some better
>> testing infrastructure.
> I tend to agree with the idea to revert it, perhaps a
Hi,
On 2021-04-21 21:21:05 -0400, Tom Lane wrote:
> What I'm doing is running the core regression tests with a single
> standby (on the same machine) and wal_consistency_checking = all.
Do you run them over replication, or sequentially by storing data into
an archive? Just curious, because its so
On Thu, Apr 22, 2021 at 1:21 PM Tom Lane wrote:
> I've also tried to reproduce on 32-bit and 64-bit Intel, without
> success. So if this is real, maybe it's related to being big-endian
> hardware? But it's also quite sensitive to $dunno-what, maybe the
> history of WAL records that have already
Andres Freund writes:
> On 2021-04-21 21:21:05 -0400, Tom Lane wrote:
>> What I'm doing is running the core regression tests with a single
>> standby (on the same machine) and wal_consistency_checking = all.
> Do you run them over replication, or sequentially by storing data into
> an archive? Ju
Hi,
On 2021-04-22 13:59:58 +1200, Thomas Munro wrote:
> On Thu, Apr 22, 2021 at 1:21 PM Tom Lane wrote:
> > I've also tried to reproduce on 32-bit and 64-bit Intel, without
> > success. So if this is real, maybe it's related to being big-endian
> > hardware? But it's also quite sensitive to $du
Andres Freund writes:
> Tom, any chance you could check if your machine repros the issue before
> these commits?
Wilco, but it'll likely take a little while to get results ...
regards, tom lane
On Thu, Apr 29, 2021 at 4:45 AM Tom Lane wrote:
> Andres Freund writes:
> > Tom, any chance you could check if your machine repros the issue before
> > these commits?
>
> Wilco, but it'll likely take a little while to get results ...
FWIW I also chewed through many megawatts trying to reproduce
Thomas Munro writes:
> FWIW I also chewed through many megawatts trying to reproduce this on
> a PowerPC system in 64 bit big endian mode, with an emulator. No
> cigar. However, it's so slow that I didn't make it to 10 runs...
Speaking of megawatts ... my G4 has now finished about ten cycles of
Hi,
On 2021-04-28 19:24:53 -0400, Tom Lane wrote:
> But I happened to notice the accumulated CPU time for the background
> processes:
>
> USER PID %CPU %MEM VSZRSS TT STAT STARTED TIME COMMAND
> tgl 19048 0.0 4.4 229952 92196 ?? Ss3:19PM 19:59.19
> post
Andres Freund writes:
> On 2021-04-28 19:24:53 -0400, Tom Lane wrote:
>> IOW, we've spent over twice as many CPU cycles shipping data to the
>> standby as we did in applying the WAL on the standby.
> I don't really know how the time calculation works on mac. Is there a
> chance it includes time s
Hi,
On 2021-04-28 20:24:43 -0400, Tom Lane wrote:
> Andres Freund writes:
> > Oh! I was about to ask how much shared buffers your primary / standby
> > have.
> Default configurations, so 128MB each.
I thought that possibly initdb would detect less or something...
I assume this is 32bit? I did
Hi,
On 2021-04-28 17:59:22 -0700, Andres Freund wrote:
> I can however say that pg_waldump on the standby's pg_wal does also
> fail. The failure as part of the backend is "invalid memory alloc
> request size", whereas in pg_waldump I get the much more helpful:
> pg_waldump: fatal: error in WAL rec
Hi,
On 2021-04-28 17:59:22 -0700, Andres Freund wrote:
> I can however say that pg_waldump on the standby's pg_wal does also
> fail. The failure as part of the backend is "invalid memory alloc
> request size", whereas in pg_waldump I get the much more helpful:
> pg_waldump: fatal: error in WAL rec
Andres Freund writes:
> I was now able to reproduce the problem again, and I'm afraid that the
> bug I hit is likely separate from Tom's.
Yeah, I think so --- the symptoms seem quite distinct.
My score so far today on the G4 is:
12 error-free regression test cycles on b3ee4c503
(plus one more
On Thu, Apr 29, 2021 at 3:14 PM Andres Freund wrote:
> To me it looks like a smaller version of the problem is present in < 14,
> albeit only when the page header is at a record boundary. In that case
> we don't validate the page header immediately, only once it's completely
> read. But we do beli
Thomas Munro writes:
> On Thu, Apr 29, 2021 at 4:45 AM Tom Lane wrote:
>> Andres Freund writes:
>>> Tom, any chance you could check if your machine repros the issue before
>>> these commits?
>> Wilco, but it'll likely take a little while to get results ...
> FWIW I also chewed through many meg
On Thu, Apr 29, 2021 at 12:24 PM Tom Lane wrote:
> Andres Freund writes:
> > On 2021-04-28 19:24:53 -0400, Tom Lane wrote:
> >> IOW, we've spent over twice as many CPU cycles shipping data to the
> >> standby as we did in applying the WAL on the standby.
>
> > I don't really know how the time cal
On Sun, May 2, 2021 at 3:16 PM Tom Lane wrote:
> That last point means that there was some hard-to-hit problem even
> before any of the recent WAL-related changes. However, 323cbe7c7
> (Remove read_page callback from XLogReader) increased the failure
> rate by at least a factor of 5, and 1d257577
On 5/3/21 7:42 AM, Thomas Munro wrote:
On Sun, May 2, 2021 at 3:16 PM Tom Lane wrote:
That last point means that there was some hard-to-hit problem even
before any of the recent WAL-related changes. However, 323cbe7c7
(Remove read_page callback from XLogReader) increased the failure
rate by
Tomas Vondra writes:
> On 5/3/21 7:42 AM, Thomas Munro wrote:
>> Hmm, yeah that does seem plausible. It would be nice to see a report
>> from any other system though. I'm still trying, and reviewing...
> FWIW I've ran the test (make installcheck-parallel in a loop) on four
> different machines
I wrote:
> I suppose that if we're unable to reproduce it on at least one other box,
> we have to write it off as hardware flakiness.
BTW, that conclusion shouldn't distract us from the very real bug
that Andres identified. I was just scraping the buildfarm logs
concerning recent failures, and I
Hi,
On 2021-05-04 15:47:41 -0400, Tom Lane wrote:
> BTW, that conclusion shouldn't distract us from the very real bug
> that Andres identified. I was just scraping the buildfarm logs
> concerning recent failures, and I found several recent cases
> that match the symptom he reported:
> [...]
> The
Hi,
On 2021-05-04 09:46:12 -0400, Tom Lane wrote:
> Yeah, I have also spent a fair amount of time trying to reproduce it
> elsewhere, without success so far. Notably, I've been trying on a
> PPC Mac laptop that has a fairly similar CPU to what's in the G4,
> though a far slower disk drive. So th
Hi,
On 2021-05-04 18:08:35 -0700, Andres Freund wrote:
> But the issue that 70b4f82a4b is trying to address seems bigger to
> me. The reason it's so easy to hit the issue is that walreceiver does <
> 8KB writes into recycled WAL segments *without* zero-filling the tail
> end of the page - which wi
On Thu, Apr 22, 2021 at 11:22 AM Stephen Frost wrote:
> On Wed, Apr 21, 2021 at 19:17 Thomas Munro wrote:
>> On Thu, Apr 22, 2021 at 8:16 AM Thomas Munro wrote:
>> ... Personally I think the right thing to do now is to revert it
>> and re-propose for 15 early in the cycle, supported with some be
On Fri, Mar 18, 2022 at 9:59 AM Thomas Munro wrote:
> I'll push 0001 today to let the build farm chew on it for a few days
> before moving to 0002.
Clearly 018_wal_optimize.pl is flapping and causing recoveryCheck to
fail occasionally, but that predates the above commit. I didn't
follow the exis
On Sun, Mar 20, 2022 at 5:36 PM Thomas Munro wrote:
> Clearly 018_wal_optimize.pl is flapping
Correction, 019_replslot_limit.pl, discussed at
https://www.postgresql.org/message-id/flat/83b46e5f-2a52-86aa-fa6c-8174908174b8%40iki.fi
.
Hi,
On Sun, Mar 20, 2022 at 05:36:38PM +1300, Thomas Munro wrote:
> On Fri, Mar 18, 2022 at 9:59 AM Thomas Munro wrote:
> > I'll push 0001 today to let the build farm chew on it for a few days
> > before moving to 0002.
>
> Clearly 018_wal_optimize.pl is flapping and causing recoveryCheck to
> f
On Mon, Mar 21, 2022 at 9:29 PM Julien Rouhaud wrote:
> So I finally finished looking at this patch. Here again, AFAICS the feature
> is
> working as expected and I didn't find any problem. I just have some minor
> comments, like for the previous patch.
Thanks very much for the review. I've a
On Thu, Mar 31, 2022 at 10:49:32PM +1300, Thomas Munro wrote:
> On Mon, Mar 21, 2022 at 9:29 PM Julien Rouhaud wrote:
> > So I finally finished looking at this patch. Here again, AFAICS the
> > feature is
> > working as expected and I didn't find any problem. I just have some minor
> > comments
On Mon, Apr 4, 2022 at 3:12 PM Julien Rouhaud wrote:
> [review]
Thanks! I took almost all of your suggestions about renaming things,
comments, docs and moving a magic number into a macro.
Minor changes:
1. Rebased over the shmem stats changes and others that have just
landed today (woo!). Th
The docs seem to be wrong about the default.
+are not yet in the buffer pool, during recovery. Valid values are
+off (the default), on and
+try. The setting try enables
+ concurrency and distance, respectively. By default, it is set to
+ try, which enabled the featu
On Fri, Apr 8, 2022 at 12:55 AM Justin Pryzby wrote:
> The docs seem to be wrong about the default.
>
> +are not yet in the buffer pool, during recovery. Valid values are
> +off (the default), on and
> +try. The setting try enables
Fixed.
> + concurrency and distance,
--Original Message-
From: Thomas Munro
Sent: Friday, April 8, 2022 10:47 AM
To: Justin Pryzby
Cc: Tomas Vondra ; Stephen Frost
; Andres Freund ; Jakub Wartak
; Alvaro Herrera ; Tomas
Vondra ; Dmitry Dolgov <9erthali...@gmail.com>;
David Steele ; pgsql-hackers
Subject: Re: WIP: WAL p
On Tue, Apr 12, 2022 at 9:03 PM Shinoda, Noriyoshi (PN Japan FSIP)
wrote:
> Thank you for developing the great feature. I tested this feature and checked
> the documentation. Currently, the documentation for the
> pg_stat_prefetch_recovery view is included in the description for the
> pg_stat_s
; Alvaro
Herrera ; Tomas Vondra
; Dmitry Dolgov <9erthali...@gmail.com>; David
Steele ; pgsql-hackers
Subject: Re: WIP: WAL prefetch (another approach)
On Tue, Apr 12, 2022 at 9:03 PM Shinoda, Noriyoshi (PN Japan FSIP)
wrote:
> Thank you for developing the great feature. I tested thi
On Thu, 7 Apr 2022 at 08:46, Thomas Munro wrote:
> With that... I've finally pushed the 0002 patch and will be watching
> the build farm.
This is a nice feature if it is safe to turn off full_page_writes.
When is it safe to do that? On which platform?
I am not aware of any released software th
On 4/12/22 15:58, Simon Riggs wrote:
> On Thu, 7 Apr 2022 at 08:46, Thomas Munro wrote:
>
>> With that... I've finally pushed the 0002 patch and will be watching
>> the build farm.
>
> This is a nice feature if it is safe to turn off full_page_writes.
>
> When is it safe to do that? On which pl
On Tue, 12 Apr 2022 at 16:41, Tomas Vondra
wrote:
>
> On 4/12/22 15:58, Simon Riggs wrote:
> > On Thu, 7 Apr 2022 at 08:46, Thomas Munro wrote:
> >
> >> With that... I've finally pushed the 0002 patch and will be watching
> >> the build farm.
> >
> > This is a nice feature if it is safe to turn o
1 - 100 of 165 matches
Mail list logo
