Re: New WAL record to detect the checkpoint redo location

2023-10-09 Thread Andres Freund
r it too, so multiple instructions can't be fully parallelized. https://uops.info/table.html documents a worst case latency of 89 cycles on cascade lake, with the division broken up into 36 uops (reducing what's available to track other in-flight instructions). It's much better on alter lake (9 cycles and 7 uops on the perf cores, 44 cycles and 4 uops on efficiency cores) and on zen 3+ (19 cycles, 2 uops). Greetings, Andres Freund

Re: New WAL record to detect the checkpoint redo location

2023-10-09 Thread Andres Freund
Hi, On 2023-10-06 13:44:55 -0400, Robert Haas wrote: > On Thu, Oct 5, 2023 at 2:34 PM Andres Freund wrote: > > If I add an unlikely around if (rechdr->xl_rmid == RM_XLOG_ID), the > > performance does improve. But that "only" brings it up to 322.406. Not sure > &

Re: remaining sql/json patches

2023-10-06 Thread Andres Freund
json_api_common_syntax > + json_returning_clause_opt > + json_value_behavior ON ERROR_P > + ')' > + { > + JsonFuncExpr *n = > makeNode(JsonFuncExpr); > + > + n->op = JSON_VALUE_OP; > + n->common = (JsonCommon *) $3; > + n->output = (JsonOutput *) $4; > + n->on_error = $5; > + n->location = @1; > + $$ = (Node *) n; > + } > + > + | JSON_VALUE '(' > + json_api_common_syntax > + json_returning_clause_opt > + json_value_behavior ON EMPTY_P > + json_value_behavior ON ERROR_P > + ')' > + { > + JsonFuncExpr *n = > makeNode(JsonFuncExpr); > + > + n->op = JSON_VALUE_OP; > + n->common = (JsonCommon *) $3; > + n->output = (JsonOutput *) $4; > + n->on_empty = $5; > + n->on_error = $8; > + n->location = @1; > + $$ = (Node *) n; > + } > ; And this. > +json_query_behavior: > + ERROR_P { $$ = > makeJsonBehavior(JSON_BEHAVIOR_ERROR, NULL, @1); } > + | NULL_P{ $$ = > makeJsonBehavior(JSON_BEHAVIOR_NULL, NULL, @1); } > + | DEFAULT a_expr{ $$ = > makeJsonBehavior(JSON_BEHAVIOR_DEFAULT, $2, @1); } > + | EMPTY_P ARRAY { $$ = > makeJsonBehavior(JSON_BEHAVIOR_EMPTY_ARRAY, NULL, @1); } > + | EMPTY_P OBJECT_P { $$ = > makeJsonBehavior(JSON_BEHAVIOR_EMPTY_OBJECT, NULL, @1); } > + /* non-standard, for Oracle compatibility only */ > + | EMPTY_P { $$ = > makeJsonBehavior(JSON_BEHAVIOR_EMPTY_ARRAY, NULL, @1); } > + ; > +json_exists_behavior: > + ERROR_P { $$ = > makeJsonBehavior(JSON_BEHAVIOR_ERROR, NULL, @1); } > + | TRUE_P{ $$ = > makeJsonBehavior(JSON_BEHAVIOR_TRUE, NULL, @1); } > + | FALSE_P { $$ = > makeJsonBehavior(JSON_BEHAVIOR_FALSE, NULL, @1); } > + | UNKNOWN { $$ = > makeJsonBehavior(JSON_BEHAVIOR_UNKNOWN, NULL, @1); } > + ; > + > +json_value_behavior: > + NULL_P { $$ = > makeJsonBehavior(JSON_BEHAVIOR_NULL, NULL, @1); } > + | ERROR_P { $$ = > makeJsonBehavior(JSON_BEHAVIOR_ERROR, NULL, @1); } > + | DEFAULT a_expr{ $$ = > makeJsonBehavior(JSON_BEHAVIOR_DEFAULT, $2, @1); } > + ; This also seems like it could use some dedup. Greetings, Andres Freund

Re: Annoying build warnings from latest Apple toolchain

2023-10-06 Thread Andres Freund
Hi, On 2023-10-05 13:37:38 -0400, Tom Lane wrote: > Andres Freund writes: > > I think you can just pass c_args directly to executable() here, I think > > adding > > c_args to default_bin_args would be a bad idea. > > Hm. IIUC that would result in an error if so

Re: Remove distprep

2023-10-06 Thread Andres Freund
think the split is useful when looking later as well. I played around with these for a bit without finding an issue. The only thing I wonder is whether we ought to keep a maintainer-clean target (as an alias to distclean), so that extensions that added things to maintainer-clean continue to work.

Re: RFC: Logging plan of the running query

2023-10-06 Thread Andres Freund
s. And presumably the interrupt would just be processed the next time interrupt processing is happening? Greetings, Andres Freund

Re: pg16: invalid page/page verification failed

2023-10-06 Thread Andres Freund
gets more than one partition? If so, this sounds like it might be the issue described here https://postgr.es/m/20230925213746.fwqauhhifjgefyzk%40alap3.anarazel.de I think at this stage the easiest fix might be just to copy the approach of calling ReleaseBulkInsertStatePin(), even though I think t

Re: New WAL record to detect the checkpoint redo location

2023-10-05 Thread Andres Freund
One thing that's notable, but not related to the patch, is that we waste a fair bit of cpu time below XLogInsertRecord() with divisions. I think they're all due to the use of UsableBytesInSegment in XLogBytePosToRecPtr/XLogBytePosToEndRecPtr. The multiplication of XLogSegNoOffsetToRecPtr() also shows. Greetings, Andres Freund

Re: Annoying build warnings from latest Apple toolchain

2023-10-05 Thread Andres Freund
initdb', >initdb_sources, >include_directories: [timezone_inc], >dependencies: [frontend_code, libpq, icu, icu_i18n], > - kwargs: default_bin_args, > + kwargs: default_bin_args + { > + 'c_args': c_args, > + }, I think you can just pass c_args directly to executable() here, I think adding c_args to default_bin_args would be a bad idea. Greetings, Andres Freund

Re: Annoying build warnings from latest Apple toolchain

2023-09-30 Thread Andres Freund
Hi, On 2023-09-29 11:11:49 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2023-09-28 22:53:09 -0400, Tom Lane wrote: > >> (Perhaps we should apply the above to HEAD alongside the meson.build fix, > >> to > >> get more test coverage?) > > > T

Re: SHARED locks barging behaviour

2023-09-30 Thread Andres Freund
ently with other locks. > > On the other hand, if nobody has complained about it in these ten years, > perhaps > it is just fine the way it is, if by design or not. I'd be very hesitant to change the behaviour at this point - the likelihood of existing workloads slowing down substantially, or even breaking due to an additional source of deadlocks, seems substantial. Greetings, Andres Freund

Re: Annoying build warnings from latest Apple toolchain

2023-09-30 Thread Andres Freund
Hi, On 2023-09-30 13:28:01 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2023-09-29 12:14:40 -0400, Tom Lane wrote: > >> Looking closer, it's only since v16 that we have export list support > >> on all officially-supported platforms. > > > Oh,

Re: Annoying build warnings from latest Apple toolchain

2023-09-30 Thread Andres Freund
Hi, On 2023-09-29 12:14:40 -0400, Tom Lane wrote: > I wrote: > > Andres Freund writes: > >> On 2023-09-28 16:46:08 -0400, Tom Lane wrote: > >>> Well, it's only important on platforms where we can't restrict > >>> libpq.so from exporting

Re: Eliminate redundant tuple visibility check in vacuum

2023-09-30 Thread Andres Freund
isible to another session when vacuum started. Greetings, Andres Freund

Re: Remove distprep

2023-09-29 Thread Andres Freund
et useful to be sure that we don't produce "untracked" build products. Do a full build, then run "full clean", then see what remains. > 88 files changed, 169 insertions(+), 409 deletions(-) It might be worthwhile to split this into a bit smaller chunks, e.g. depending on perl, bison, flex, and then separately the various makefile bits that are all over the tree. Greetings, Andres Freund

Re: how to manage Cirrus on personal repository

2023-09-29 Thread Andres Freund
ory and only > activate Cirrus on that one, and push there on demand. You already can control which platforms are run via commit messages, fwiw. By adding, e.g.: ci-os-only: linux, macos Greetings, Andres Freund

Re: Annoying build warnings from latest Apple toolchain

2023-09-29 Thread Andres Freund
Hi, On 2023-09-28 22:53:09 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2023-09-28 19:20:27 -0700, Andres Freund wrote: > >> Thus the easiest fix looks to be to use this: > >> - export_fmt = '-exported_symbols_list=@0@' > >> + export_fmt =

Re: Annoying build warnings from latest Apple toolchain

2023-09-28 Thread Andres Freund
Hi, On 2023-09-28 19:20:27 -0700, Andres Freund wrote: > Thus the easiest fix looks to be to use this: > > diff --git a/meson.build b/meson.build > index 5422885b0a2..16a2b0f801e 100644 > --- a/meson.build > +++ b/meson.build > @@ -224,7 +224,7 @@ elif h

Re: Annoying build warnings from latest Apple toolchain

2023-09-28 Thread Andres Freund
Hi, On 2023-09-28 19:17:37 -0400, Tom Lane wrote: > I wrote: > > Andres Freund writes: > >> I think right now it doesn't work as-is on sonoma, because apple decided to > >> change the option syntax, which is what causes the -e warning below, so the >

Re: Annoying build warnings from latest Apple toolchain

2023-09-28 Thread Andres Freund
Hi, On 2023-09-28 16:46:08 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2023-09-27 16:52:44 -0400, Tom Lane wrote: > >> I think it doesn't, as long as all the relevant build targets > >> write their dependencies with "frontend_code" before &qu

Re: Annoying build warnings from latest Apple toolchain

2023-09-28 Thread Andres Freund
7;s quite annoying that apple is changing things option syntax. > (I also tried with a more recent meson version, 1.1.1, with > the same result.) Looks like you need 1.2 for the new clang / ld output... Apparently apple's linker changed the format of its version output :/. Greetings, Andres Freund

Re: pg_stat_get_activity(): integer overflow due to (int) * (int) for MemoryContextAllocHuge()

2023-09-27 Thread Andres Freund
Hi, On 2023-09-28 07:53:45 +0900, Michael Paquier wrote: > On Wed, Sep 27, 2023 at 10:29:25AM -0700, Andres Freund wrote: > > Frankly, it seems like a quite bad idea to have such a high limit for > > pgstat_track_activity_query_size. The overhead such a high value has will >

Re: Eager page freeze criteria clarification

2023-09-27 Thread Andres Freund
On 2023-09-27 19:09:41 -0400, Melanie Plageman wrote: > On Wed, Sep 27, 2023 at 3:25 PM Robert Haas wrote: > > > > On Wed, Sep 27, 2023 at 12:34 PM Andres Freund wrote: > > > One way to deal with that would be to not track the average age in > > > LSN-differenc

Re: Eager page freeze criteria clarification

2023-09-27 Thread Andres Freund
hings onto early pages - it e.g. can slow down concurrent copy workloads by 3-4x due to contention in the FSM - and that it has more size classes than necessary, but I don't think just closing frozen pages against further insertions of small tuples will cause its own set of issues. I think at the very least there'd need to be something causing pages to reopen once the aggregate unused space in the table reaches some threshold. Greetings, Andres Freund

Re: Eager page freeze criteria clarification

2023-09-27 Thread Andres Freund
Hi, On 2023-09-27 13:14:41 -0700, Peter Geoghegan wrote: > On Wed, Sep 27, 2023 at 1:03 PM Andres Freund wrote: > > I suspect that medium term the better approach would be to be much more > > aggressive about setting all-visible, including as part of page-level > > visibilit

Re: Streaming I/O, vectored I/O (WIP)

2023-09-27 Thread Andres Freund
gStreamingRead *pgsr); > > Do we need to expose pg_streaming_read_prefetch()? It's only used in the WAL > replay prefetching patch, and only after calling pg_streaming_read_reset(). > Could pg_streaming_read_reset() call pg_streaming_read_prefetch() directly? > Is there any need to "reset" the stream, without also starting prefetching? Heh, I think this is a discussion Thomas I were having before... Greetings, Andres Freund

Re: Eager page freeze criteria clarification

2023-09-27 Thread Andres Freund
we adopt some > proposal that does a lot more opportunistic freezing than we do > currently. Because then the problem just wouldn't come up nearly as > much as it does now. One patch can't fix everything, and shouldn't > try. Agreed. Greetings, Andres Freund

Re: Eager page freeze criteria clarification

2023-09-27 Thread Andres Freund
Hi, On 2023-09-27 10:25:00 -0700, Peter Geoghegan wrote: > On Wed, Sep 27, 2023 at 10:01 AM Andres Freund wrote: > > On 2023-09-26 09:07:13 -0700, Peter Geoghegan wrote: > > I don't think doing this on a system wide basis with a metric like #unfrozen > > pages is a go

Re: pg_stat_get_activity(): integer overflow due to (int) * (int) for MemoryContextAllocHuge()

2023-09-27 Thread Andres Freund
e bad idea to have such a high limit for pgstat_track_activity_query_size. The overhead such a high value has will surprise people... Greetings, Andres Freund

Re: Set enable_seqscan doesn't take effect?

2023-09-27 Thread Andres Freund
prior research. Often even sending the same question to multiple lists. It should not be hard to find an explanation for the behaviour you see here. pgsql-hackers isn't a "do my work for me service". We're hacking on postgres. It's fine to occasionally ask for direction, but you're very clearly exceeding that. Greetings, Andres Freund

Re: Eager page freeze criteria clarification

2023-09-27 Thread Andres Freund
Hi, On 2023-09-26 09:07:13 -0700, Peter Geoghegan wrote: > On Tue, Sep 26, 2023 at 8:19 AM Andres Freund wrote: > > However, I'm not at all convinced doing this on a system wide level is a > > good > > idea. Databases do often contain multiple types of workloads at t

Re: Eager page freeze criteria clarification

2023-09-27 Thread Andres Freund
Hi, (I wrote the first part of the email before Robert and I chatted on a call, I left it in the email for posterity) On 2023-09-26 13:49:32 -0400, Robert Haas wrote: > On Tue, Sep 26, 2023 at 11:11 AM Andres Freund wrote: > > As long as the most extreme cases are prevented, unne

Re: Eager page freeze criteria clarification

2023-09-26 Thread Andres Freund
ely in a database that has the bulk of its size in archival partitions but has lots of unfrozen data in an active partition. And databases have often loads of data that's going to change frequently / isn't long lived, and we don't want to super aggressively freeze that, just because it's a large portion of the data. Greetings, Andres Freund

Re: Eager page freeze criteria clarification

2023-09-26 Thread Andres Freund
Hi, On 2023-09-25 14:45:07 -0400, Robert Haas wrote: > On Fri, Sep 8, 2023 at 12:07 AM Andres Freund wrote: > > > Downthread, I proposed using the RedoRecPtr of the latest checkpoint > > > rather than the LSN of the previou vacuum. I still like that idea. > > &g

Re: Performance degradation on concurrent COPY into a single relation in PG16.

2023-09-25 Thread Andres Freund
Hi, On 2023-09-25 12:48:30 -0700, Andres Freund wrote: > On 2023-09-25 15:42:26 -0400, Tom Lane wrote: > > I just did a git bisect run to discover when the failure documented > > in bug #18130 [1] started. And the answer is commit 82a4edabd. > > Now, it's pretty obvio

Re: Performance degradation on concurrent COPY into a single relation in PG16.

2023-09-25 Thread Andres Freund
provoke the bug... A trigger might prevent using the multi-insert API, which would lead to different execution paths... Greetings, Andres Freund

Re: DROP DATABASE is interruptible

2023-09-25 Thread Andres Freund
Hi, On 2023-09-25 01:48:31 +0100, Peter Eisentraut wrote: > I noticed that this patch set introduced this pg_dump test: > > On 12.07.23 03:59, Andres Freund wrote: > > + 'CREATE DATABASE invalid...' => { > > + create_order => 1, > > +

Re: Disabling Heap-Only Tuples

2023-09-21 Thread Andres Freund
ation extension "covers" - the reason that we need to pin the to-be-extended-pages is to prevent concurrent scans from reading "new" blocks while the extension is in progress, as otherwise such a buffer can be dirtied and written out, potentially leading to lost writes and other fun issues. But with the shared smgrrel, we can store the size-currently-being-extended-to separately from the filesystem size. If it's not allowed to read the block range covered by those blocks into s_b, the race doesn't exist anymore. Greetings, Andres Freund

Re: Disabling Heap-Only Tuples

2023-09-21 Thread Andres Freund
Hi, On 2023-09-19 14:50:13 -0400, Robert Haas wrote: > On Tue, Sep 19, 2023 at 12:56 PM Andres Freund wrote: > > Yea, a setting like what's discussed here seems, uh, not particularly useful > > for achieving the goal of compacting tables. I don't think guiding this &g

Re: GenBKI emits useless open;close for catalogs without rows

2023-09-21 Thread Andres Freund
e care about doing bootstrap mode interactively? We could probably > remove that. Heh, yea, that's pretty pointless. Greetings, Andres Freund

Re: Disabling Heap-Only Tuples

2023-09-19 Thread Andres Freund
Hi, On 2023-09-19 19:33:22 +0200, Matthias van de Meent wrote: > On Tue, 19 Sept 2023 at 18:56, Andres Freund wrote: > > > > Hi, > > > > On 2023-09-19 18:30:44 +0200, Alvaro Herrera wrote: > > > This makes me think that maybe the logic needs to be a little mor

Re: Disabling Heap-Only Tuples

2023-09-19 Thread Andres Freund
ently determined. But I think > > for most people in most situations, it will be a struggle. Indeed. I think it'd often just explode table and index sizes, because HOT pruning won't be able to make usable space in pages anymore (due to dead items). Greetings, Andres Freund

Re: Move bki file pre-processing from initdb to bootstrap

2023-09-18 Thread Andres Freund
s also lots of projects using postgres. The main reason I'd like to move this infrastructure to the backend is that I really would like to get rid of single user mode. It adds complications all over, it's barely tested, pointlessly hard to use. I wrote a rough prototype of that a while back: https://postgr.es/m/20220220214439.bhc35hhbaub6dush%40alap3.anarazel.de Greetings, Andres Freund

Re: XLog size reductions: smaller XLRec block header for PG17

2023-09-18 Thread Andres Freund
t encodings, just to eek out a bit more space savings. The increase in code complexity IMO makes it a bad tradeoff. Greetings, Andres Freund

Re: Sync scan & regression tests

2023-09-18 Thread Andres Freund
> dates to Tomas' commit 7db0cd214 of 2021-01-17, and chipmunk > > passed many times after that. Might be worth bisecting in > > the interval where chipmunk wasn't reporting? > > I bisected it to this: > > commit 82a4edabd272f70d044faec8cf7fd1eab92d9991 (HEAD)

Re: [PoC] Improve dead tuple storage for lazy vacuum

2023-09-15 Thread Andres Freund
Hi, On 2023-08-28 23:43:22 +0900, Masahiko Sawada wrote: > I've attached v42 patch set. I improved tidstore regression test codes > in addition of imcorporating the above comments. Why did you need to disable the benchmark module for CI? Greetings, Andres Freund

Re: Performance degradation on concurrent COPY into a single relation in PG16.

2023-09-15 Thread Andres Freund
f7fd1eab92d9991 is the first bad commit > commit 82a4edabd272f70d044faec8cf7fd1eab92d9991 > Author: Andres Freund > Date: Mon Aug 14 09:54:03 2023 -0700 > > hio: Take number of prior relation extensions into account > > but I imagine the problem is that the patch's more aggressiv

Re: Is it possible to change wal_level online

2023-09-14 Thread Andres Freund
Hi, On September 14, 2023 6:21:59 AM PDT, Euler Taveira wrote: >On Thu, Sep 14, 2023, at 7:05 AM, Andy Fan wrote: >> Currently it is complained that wal_level changes require an instance >> restart, I'm not familiar with this stuff so far and I didn't get any good >> information from searching

Re: Have better wording for snapshot file reading failure

2023-09-14 Thread Andres Freund
t exist", similar to what we report for unknown tables etc. Arguably changing the errcode to ERRCODE_UNDEFINED_OBJECT would make this more precise? > This new suggestion is only for HEAD. I've reverted a0d87bc & co for > now. I think there's really no reason to backpatch this, so that makes sense to me. Greetings, Andres Freund

Re: BufferUsage counters' values have changed

2023-09-13 Thread Andres Freund
Hi, On 2023-09-13 11:59:39 -0700, Andres Freund wrote: > Running the attached patch through CI, planning to push after that succeeds, > unless somebody has a comment? And pushed. Thanks Karina for the report and fix! Greetings, Andres Freund

Re: Have better wording for snapshot file reading failure

2023-09-13 Thread Andres Freund
Hi, On 2023-09-13 19:07:24 -0700, Andres Freund wrote: > On 2023-09-14 10:33:33 +0900, Michael Paquier wrote: > > On Wed, Sep 13, 2023 at 01:19:38PM +0200, Daniel Gustafsson wrote: > > > +1. This errmsg is already present so it eases the translation burden as > > > we

Re: Have better wording for snapshot file reading failure

2023-09-13 Thread Andres Freund
Hi, On 2023-09-14 10:33:33 +0900, Michael Paquier wrote: > On Wed, Sep 13, 2023 at 01:19:38PM +0200, Daniel Gustafsson wrote: > > +1. This errmsg is already present so it eases the translation burden as > > well. > > I was thinking about doing only that on HEAD, but there is an argument > that o

Re: BufferUsage counters' values have changed

2023-09-13 Thread Andres Freund
7;t visible anywhere, iirc. Thus I added a test to pg_stat_statements. Afaict it should be stable? Running the attached patch through CI, planning to push after that succeeds, unless somebody has a comment? Greetings, Andres Freund >From ae7b170e9032af0fd257a39953fe0b6ef7e9637c Mon Sep 17

Re: [PATCH] Add native windows on arm64 support

2023-09-13 Thread Andres Freund
mages would be some one-off work, but after that... Greetings, Andres Freund

Re: BufferUsage counters' values have changed

2023-09-13 Thread Andres Freund
is 3, this is called 1 time) [2] > > I think [0] is the same path as [1] but [2] is new. 'fsm extends' > wasn't counted in local_blks_written in PG_15. Calling > ExtendBufferedRelTo() from fsm_extend() caused 'fsm extends' to be > counted in local_blks_written. I am not sure which one is correct. I think it's correct to count the fsm writes here. The pg_stat_statement columns aren't specific to the main relation for or such... If anything it was a bug to not count them before. Thanks for looking into this. Greetings, Andres Freund

Re: Eliminate redundant tuple visibility check in vacuum

2023-09-13 Thread Andres Freund
int8 *htsv, >PruneState *prstate); Hm, do we really want to pass this explicitly to a bunch of functions? Seems like it might be better to either pass the PruneResult around or to have a pointer in PruneState? > /* >* The criteria for counting a tuple as live in this block need > to > @@ -1682,7 +1664,7 @@ retry: >* (Cases where we bypass index vacuuming will violate this > optimistic >* assumption, but the overall impact of that should be > negligible.) >*/ > - switch (res) > + switch ((HTSV_Result) presult.htsv[offnum]) > { > case HEAPTUPLE_LIVE: I think we should assert that we have a valid HTSV_Result here, i.e. not -1. You could wrap the cast and Assert into an inline funciton as well. Greetings, Andres Freund

Re: Query execution in Perl TAP tests needs work

2023-09-12 Thread Andres Freund
s substantially. Famous last words, but it seems like that it should be quite doable to add that to psql and use it in Cluster->{psql,safe_psql,poll_query_until}? There might be a few cases that expect the full connection error message, but I can't imagine that to be too many? Greetings, Andres Freund

Re: Cirrus-ci is lowering free CI cycles - what to do with cfbot, etc?

2023-09-12 Thread Andres Freund
r0m6.596s sys 0m8.458s Daniel's (probing every 50ms with PQping()): cassert: real0m3.347s user0m8.373s sys 0m8.354s optimized: real0m2.527s user0m6.156s sys 0m8.315s My patch increased user/sys time a bit (likely due to a higher number of futile psql forks),

Re: broken master regress tests

2023-09-11 Thread Andres Freund
(Not sure why this happened.) > > So I think injecting --no-locale to the initdb line that creates the > template is a better approach; something like the attached. Makes sense, thanks for taking care of this. Greetings, Andres Freund

Re: Eager page freeze criteria clarification

2023-09-08 Thread Andres Freund
Hi, On 2023-09-07 22:29:04 -0700, Peter Geoghegan wrote: > On Thu, Sep 7, 2023 at 9:45 PM Andres Freund wrote: > > I.e. setting an, otherwise unmodified, page all-visible won't trigger an FPI > > if checksums are disabled, but will FPI with checksums enabled. I thi

Re: Eager page freeze criteria clarification

2023-09-07 Thread Andres Freund
Hi, On 2023-09-07 21:45:22 -0700, Andres Freund wrote: > In contrast to that, freezing will almost always trigger an FPI (except for > empty pages, but we imo ought to stop setting empty pages all frozen [1]). > > > Yep, a quick experiment confirms that: > > DROP TABLE IF

Re: Eager page freeze criteria clarification

2023-09-07 Thread Andres Freund
a recent FPI, and doesn't need pruning, but which > can be marked all-visible, freezing it is a potentially more > significant cost, because marking the buffer all-visible doesn't force > a new FPI, and freezing does. +1 One thing that bothers in this area when using will-trigger-FPI style logic, is that it makes checksums=on/off have behavioural impact. If we e.g. make setting all-visible conditional on not triggering an FPI, there will be plenty workloads where a checksums=off system will get an index only scan, but a checksums=on system won't. Greetings, Andres Freund

Re: Eager page freeze criteria clarification

2023-09-07 Thread Andres Freund
132752 27655515 [7.11%] 361110616 [92.89%]388766131 [100%] In realistic tables, where rows are wider than a single int, FPI_FOR_HINT dominates even further, as the FREEZE_PAGE would be smaller if there weren't 226 tuples o

Re: Eager page freeze criteria clarification

2023-09-07 Thread Andres Freund
Hi, On 2023-09-06 10:35:17 -0400, Robert Haas wrote: > On Wed, Sep 6, 2023 at 1:09 AM Andres Freund wrote: > > Yea, it'd be useful to have a reasonably approximate wall clock time for the > > last modification of a page. We just don't have infrastructure for > >

Re: Eager page freeze criteria clarification

2023-09-05 Thread Andres Freund
time spent vacuuming (freezing less aggressively might increase the number of vacuums due to anti-wrap vacuums, at the same time, freezing too aggressively could lead to vacuums taking too long) 4) throughput of the workload (to see potential regressions due to vacuuming overhead) 5) for transactional workloads: p99 latency (to see if vacuuming increases commit latency and such, just using average tends to hide too much) Greetings, Andres Freund

Re: Initdb-time block size specification

2023-09-05 Thread Andres Freund
Hi, On 2023-09-05 21:52:18 +0200, Hannu Krosing wrote: > Something I also asked at this years Unconference - Do we currently > have Build Farm animals testing with different page sizes ? You can check that yourself as easily as anybody else. Greetings, Andres Freund

Re: Improve heapgetpage() performance, overhead from serializable

2023-09-05 Thread Andres Freund
Hi, On 2023-09-05 14:42:57 +0700, John Naylor wrote: > On Mon, Jul 17, 2023 at 9:58 PM Andres Freund wrote: > > > FWIW, there's more we can do, with some hacky changes I got the time down > to > > 273.261, but the tradeoffs start to be a bit more complicated. And >

Re: Query execution in Perl TAP tests needs work

2023-08-30 Thread Andres Freund
g for IO should be easy... > 2b. Maybe give psql or a new libpq-wrapper a new low level stdio/pipe > protocol that is more fun to talk to from Perl/machines? That does also seem promising - a good chunk of the complexity around some of the IPC::Run uses is that we end up parsing psql input/output... Greetings, Andres Freund

Re: initdb caching during tests

2023-08-25 Thread Andres Freund
Hi, On 2023-08-25 09:00:24 +0200, Daniel Gustafsson wrote: > > On 25 Aug 2023, at 07:50, Thomas Munro wrote: > > > > On Fri, Aug 25, 2023 at 10:10 AM Andres Freund wrote: > >> Let's see what the buildfarm says - it's not inconceivable that it'll

Re: Format list of catalog files in makefile vertically

2023-08-25 Thread Andres Freund
Hi, On August 25, 2023 9:12:51 AM EDT, Peter Eisentraut wrote: >I propose to reformat the catalog lists in src/backend/catalog/Makefile to be >more vertical, one per line. This makes it easier to keep that list in sync >with src/include/catalog/meson.build, and visually compare both lists.

Re: initdb caching during tests

2023-08-24 Thread Andres Freund
Hi, On 2023-08-23 10:10:31 +0200, Daniel Gustafsson wrote: > > On 23 Aug 2023, at 03:17, Andres Freund wrote: > > On 2023-08-22 23:47:24 +0200, Daniel Gustafsson wrote: > > >> My only small gripe is that I keep thinking about template databases for > >> CREA

Re: Cirrus-ci is lowering free CI cycles - what to do with cfbot, etc?

2023-08-24 Thread Andres Freund
Hi, On 2023-08-22 23:58:33 -0700, Andres Freund wrote: > To make that possible, we need to make the compute resources for CI > configurable on a per-repository basis. After experimenting with a bunch of > ways to do that, I got stuck on that for a while. But since today we have >

Re: meson uses stale pg_config_paths.h left over from make

2023-08-23 Thread Andres Freund
to debug issues that creates. I've spent quite a bit of time helping people to debug such issues. Greetings, Andres Freund

Re: Cirrus-ci is lowering free CI cycles - what to do with cfbot, etc?

2023-08-23 Thread Andres Freund
Hi, On 2023-08-23 18:32:26 -0400, Tom Lane wrote: > Andres Freund writes: > > There are other potential uses for libpq in pg_regress though - I'd e.g. > > like > > to have a "monitoring" session open, which we could use to detect that the > > server c

Re: Cirrus-ci is lowering free CI cycles - what to do with cfbot, etc?

2023-08-23 Thread Andres Freund
Hi, On 2023-08-23 17:55:53 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2023-08-23 17:02:51 -0400, Tom Lane wrote: > >> ... unless we hit problems with, say, a different default port number or > >> socket path compiled into one copy vs. the other? That seem

Re: Cirrus-ci is lowering free CI cycles - what to do with cfbot, etc?

2023-08-23 Thread Andres Freund
Hi, On 2023-08-23 23:55:15 +0300, Nazir Bilal Yavuz wrote: > On Wed, 23 Aug 2023 at 09:58, Andres Freund wrote: > > I'm hoping to push this fairly soon, as I'll be on vacation the last week of > > August. I'll be online intermittently though, if there are issu

Re: Cirrus-ci is lowering free CI cycles - what to do with cfbot, etc?

2023-08-23 Thread Andres Freund
Hi, On 2023-08-23 17:02:51 -0400, Tom Lane wrote: > Daniel Gustafsson writes: > > On 23 Aug 2023, at 21:22, Andres Freund wrote: > >> I think there's more effective ways to make this cheaper. The basic thing > >> would be to use libpq instead of forking of psql

Re: Cirrus-ci is lowering free CI cycles - what to do with cfbot, etc?

2023-08-23 Thread Andres Freund
Hi, On 2023-08-23 14:48:26 +0200, Daniel Gustafsson wrote: > > On 23 Aug 2023, at 08:58, Andres Freund wrote: > > > I'm hoping to push this fairly soon, as I'll be on vacation the last week of > > August. I'll be online intermittently though, if there ar

Re: Cirrus-ci is lowering free CI cycles - what to do with cfbot, etc?

2023-08-22 Thread Andres Freund
Hi, On 2023-08-07 19:15:41 -0700, Andres Freund wrote: > As some of you might have seen when running CI, cirrus-ci is restricting how > much CI cycles everyone can use for free (announcement at [1]). This takes > effect September 1st. > > This obviously has consequences both for i

Re: initdb caching during tests

2023-08-22 Thread Andres Freund
hink this is committable as is. Cool. Planning to do that tomorrow. We can easily extend / adjust this later, it just affects testing infrastructure. Greetings, Andres Freund

Re: ci: Improve macos startup using a cached macports installation

2023-08-19 Thread Andres Freund
On 2023-08-19 11:47:33 -0700, Andres Freund wrote: > Given how significant an improvement this is in test time, and the limited > blast radius, I am planning to push this soon, unless somebody opposes that > soon. And done.

Re: postgres_fdw: wrong results with self join + enable_nestloop off

2023-08-19 Thread Andres Freund
that that might still be the right path. > BTW: commit 9e9931d2b removed the restriction on the call to the hook > extensions, so you might want to back-patch it. Citus is an extension, not a fork, there's not really a way to just backpatch a random commit. > Though, I think it would be better if the hook was well implemented from the > beginning. Sure, but that's neither here nor there. Greetings, Andres Freund

Re: CREATE FUNCTION ... SEARCH { DEFAULT | SYSTEM | SESSION }

2023-08-19 Thread Andres Freund
Hi, On 2023-08-14 12:25:30 -0700, Jeff Davis wrote: > On Sat, 2023-08-12 at 11:25 -0700, Andres Freund wrote: > > > > I'm not sure that anything based, directly or indirectly, on > > search_path > > really is a realistic way to get there. > > Can you expl

Re: ci: Improve macos startup using a cached macports installation

2023-08-19 Thread Andres Freund
Hi, On 2023-08-05 13:25:39 -0700, Andres Freund wrote: > We have some issues with CI on macos and windows being too expensive (more on > that soon in a separate email). For macos most of the obviously wasted time is > spent installing packages with homebrew. Even with the package downlo

Re: Remove distprep

2023-08-17 Thread Andres Freund
On 2023-08-18 10:11:12 +0900, Michael Paquier wrote: > On Wed, Aug 09, 2023 at 07:38:40AM -0700, Andres Freund wrote: > > On 2023-08-09 16:25:28 +0200, Christoph Berg wrote: > >> Understood, I was just pointing out there are more types of generated > >> files in there.

Re: meson: pgxs Makefile.global differences

2023-08-17 Thread Andres Freund
think that's feasible. It was a fair bit of work to get the most important contents to match, while skipping lots of things that are primarily relevant for building the server (which isn't relevant for pgxs). That said, in this specific case, I agree, we should likely emit -Wall to Makefile.global in meson as well. Greetings, Andres Freund PS: I don't have and...@anarazel.dev , just .de :)

Re: Rename ExtendedBufferWhat in 16?

2023-08-16 Thread Andres Freund
On 2023-08-17 11:31:27 +1200, Thomas Munro wrote: > On Thu, Aug 17, 2023 at 10:49 AM Andres Freund wrote: > > On 2023-08-12 12:29:05 +1200, Thomas Munro wrote: > > > Commit 31966b15 invented a way for functions dealing with relation > > > extension to accept a Rel

Re: Performance degradation on concurrent COPY into a single relation in PG16.

2023-08-16 Thread Andres Freund
On 2023-08-16 13:15:46 +0200, Alvaro Herrera wrote: > Since the wins from this patch were replicated and it has been pushed, I > understand that this open item can be marked as closed, so I've done > that. Thanks!

Re: Rename ExtendedBufferWhat in 16?

2023-08-16 Thread Andres Freund
o propose for 17, which needs > one of these too, hence desire to generalise. But if we rename them > in 17, then AM authors, who are likely to discover and make use of > this interface, would have to use different names for 16 and 17. Makes sense to me. Greetings, Andres Freund

Re: pg_logical_emit_message() misses a XLogFlush()

2023-08-15 Thread Andres Freund
lso object to making existing invocations flush WAL in HEAD. I do not at all object to adding a parameter that indicates flushing, or a separate function to do so. The latter might be better, because it allows you to flush a group of messages, rather than a single one. Greetings, Andres Freund

Re: pg_logical_emit_message() misses a XLogFlush()

2023-08-15 Thread Andres Freund
Hi, On 2023-08-16 03:20:53 +0200, Tomas Vondra wrote: > On 8/16/23 02:33, Andres Freund wrote: > > Hi, > > > > On 2023-08-16 06:58:56 +0900, Michael Paquier wrote: > >> On Tue, Aug 15, 2023 at 11:37:32AM +0200, Tomas Vondra wrote: > >>> Shouldn'

Re: pg_logical_emit_message() misses a XLogFlush()

2023-08-15 Thread Andres Freund
ear to me that flushing would be the right behaviour if it weren't released, but it certainly doesn't seem right to make such a change in a minor release. Greetings, Andres Freund

Re: run pgindent on a regular basis / scripted manner

2023-08-12 Thread Andres Freund
It's a somewhat annoying task though, find all the typedefs, add them to the right place in the file (we have an out of order entry right now). I think a script that *adds* (but doesn't remove) local typedefs would make this less painful. Greetings, Andres Freund

Re: A failure in 031_recovery_conflict.pl on Debian/s390x

2023-08-12 Thread Andres Freund
ery_conflict.pl passes even after I make ConditionalLockBufferForCleanup() fail 100%. Greetings, Andres Freund

Re: Performance degradation on concurrent COPY into a single relation in PG16.

2023-08-12 Thread Andres Freund
h that case, because of the MAX_BUFFERED_* limits in copyfrom.c, but now it's more common. But that probably ought to be done only HEAD, not 16. A related oddity: RelationExtensionLockWaiterCount()->LockWaiterCount() uses an exclusive lock on the lock partition - which seems not at all nece

Re: Fix pg_stat_reset_single_table_counters function

2023-08-12 Thread Andres Freund
> statistics after resetting the stats. What about updating the table and checking the update count is reset? That'd not be reset by autovacuum. Greetings, Andres Freund

Re: CREATE FUNCTION ... SEARCH { DEFAULT | SYSTEM | SESSION }

2023-08-12 Thread Andres Freund
are written in just a few languages. (*) Obviously the one thing that doesn't work for is use of EXECUTE in plpgsql and similar constructs elsewhere. I'm not sure there's much that can be done to make that safe, but it's worth thinking about more. Greetings, Andres Freund

Re: run pgindent on a regular basis / scripted manner

2023-08-11 Thread Andres Freund
Hi, On 2023-08-11 20:11:34 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2023-08-11 18:30:02 -0400, Tom Lane wrote: > >> +1 for including this in CI tests > > > I didn't even mean CI - I meant 'make check-world' / 'meson test'. Which o

Re: run pgindent on a regular basis / scripted manner

2023-08-11 Thread Andres Freund
Hi, On 2023-08-11 18:30:02 -0400, Tom Lane wrote: > Peter Geoghegan writes: > > On Fri, Aug 11, 2023 at 2:25 PM Andres Freund wrote: > >> We could a test that fails when there's some mis-indented code. That seems > >> like it might catch things earlier? > >

Re: run pgindent on a regular basis / scripted manner

2023-08-11 Thread Andres Freund
We could a test that fails when there's some mis-indented code. That seems like it might catch things earlier? Greetings, Andres Freund

<    4   5   6   7   8   9   10   11   12   13   >