Re: run pgindent on a regular basis / scripted manner

2023-08-11 Thread Peter Geoghegan
ess that I can't do that anymore. Hopefully maintaining the typedefs.list file isn't as inconvenient as it once seemed to me to be. -- Peter Geoghegan

Re: run pgindent on a regular basis / scripted manner

2023-08-11 Thread Peter Geoghegan
all that simple. Don't typedefs need to be downloaded from some other blessed buildfarm animal? -- Peter Geoghegan

Re: AssertLog instead of Assert in some places

2023-08-11 Thread Peter Geoghegan
information that is output should be optimized for the "duh, of course" scenarios. Not to an absurd degree, mind you. But the output shouldn't be too verbose. Ideally there'd be a still fairly straightforward way of getting extra information, for the cases where debugging is likely to take a few minutes, and require real focus. The extra work in those other cases is relatively insignificant, because the "startup costs" are relatively large -- a little extra indirection (though only a little) can't hurt too much. -- Peter Geoghegan

Re: run pgindent on a regular basis / scripted manner

2023-08-11 Thread Peter Geoghegan
dfarm member to enforce that rule, too? -- Peter Geoghegan

Re: AssertLog instead of Assert in some places

2023-08-11 Thread Peter Geoghegan
> restart-after-crash by default, and not start new tests once the server has > crashed and do a waitpid(postmaster, WNOHANG) after each failing test, to see > if the reason the test failed is that the backend died. +1 -- Peter Geoghegan

Re: AssertLog instead of Assert in some places

2023-08-11 Thread Peter Geoghegan
elf. But now that I've thought about it a little more, I think that such an approach might end up being overused. If you're going to add what amounts to a "can't happen" ERROR then you should really be obligated to write a halfway reasonable error message. As I said, you should

Re: AssertLog instead of Assert in some places

2023-08-11 Thread Peter Geoghegan
is quite different to what Ashutosh has proposed, though, since it would still look like a conventional assertion failure on assert-enabled builds.) -- Peter Geoghegan

Re: Use of additional index columns in rows filtering

2023-08-09 Thread Peter Geoghegan
that seem just about unavoidable in any future world. (Come to think of it, the biggest and most durable advantage for index filters is probably just how general they are, which I do mention.) -- Peter Geoghegan General principles for index quals == Tomas Vondr

Re: Use of additional index columns in rows filtering

2023-08-09 Thread Peter Geoghegan
elease or two. As I said, I mostly just care about having the right general principles in place at this point. > OTOH if there was some facility to decide if a qual is "safe" to be > executed on the index tuple, that'd be nice. But as I already said, I > see it more as an additional optimization, as it only applies to a > subset of cases. I'm happy to go with that. -- Peter Geoghegan

Re: Use of additional index columns in rows filtering

2023-08-09 Thread Peter Geoghegan
es are very common -- it's surprising that nbtree can't already handle them directly. I should have thought of this myself, long ago, but it took your patch to force me to think about it. The problem with simple inequalities was "hiding in plain sight" for a very long time. Could there be anything else like that? -- Peter Geoghegan

Re: pg15: reltuples stuck at -1 after pg_upgrade and VACUUM

2023-08-09 Thread Peter Geoghegan
_upgrade. The ANALYZE should update this. Right. VACUUM is sometimes much less efficient than just using ANALYZE to establish an initial reltuples. Other times it is much less accurate. I can't see any argument for opting to use VACUUM instead of ANALYZE after an upgrade. -- Peter Geoghegan

Re: pg15: reltuples stuck at -1 after pg_upgrade and VACUUM

2023-08-08 Thread Peter Geoghegan
> are left in the "never have been vacuumed" state. But -1 isn't the "never have been vacuumed" state, exactly. At best it is the "never been vacuumed or analyzed" state. -- Peter Geoghegan

Re: Use of additional index columns in rows filtering

2023-08-08 Thread Peter Geoghegan
d what the issue is - the example seems > like a run-of-the-mill case of poor estimate / costing). I'm not trying to impose any particular interpretation here. It's early in the cycle, and my questions are mostly exploratory. I'm still trying to develop my own understanding of the trade-offs in this area. -- Peter Geoghegan

Re: Use of additional index columns in rows filtering

2023-08-08 Thread Peter Geoghegan
to true index quals. I think I'd like to have a go at it myself. To some degree I'm trying to get a sense of how much that'd help you. -- Peter Geoghegan

Re: Use of additional index columns in rows filtering

2023-08-08 Thread Peter Geoghegan
On Tue, Aug 8, 2023 at 11:36 AM Peter Geoghegan wrote: > Assuming that that happens, then it immediately gives index scans a > huge advantage over bitmap index scans. At that point it seems > important to describe (in high level terms) where it is that the > advantage is innate, an

Re: Use of additional index columns in rows filtering

2023-08-08 Thread Peter Geoghegan
going to insist on it, or anything like that. It just seems like something that'll add a great deal of value over what you have already. -- Peter Geoghegan

Re: Use of additional index columns in rows filtering

2023-08-08 Thread Peter Geoghegan
On Mon, Aug 7, 2023 at 9:21 PM Peter Geoghegan wrote: > I see that get_op_btree_interpretation() will treat != as a kind of > honorary member of an opfamily whose = operator has our != operator as > its negator. Perhaps we should be finding a way to pass != quals into > the index AM

Re: Use of additional index columns in rows filtering

2023-08-08 Thread Peter Geoghegan
els artificial. I suggest that you focus on that aspect, since it has the potential to make what is already a compelling patch into a much more compelling patch. -- Peter Geoghegan

Re: Use of additional index columns in rows filtering

2023-08-07 Thread Peter Geoghegan
On Mon, Aug 7, 2023 at 3:18 PM Peter Geoghegan wrote: > Even my patch cannot always make SAOP clauses into index quals. There > are specific remaining gaps that I hope that your patch will still > cover. The simplest example is a similar NOT IN() inequality, like > this: > &g

Re: Use of additional index columns in rows filtering

2023-08-07 Thread Peter Geoghegan
But that's not going to happen for 17, and doesn't seem all that helpful with a query like this in any case. But it does suggest an argument in favor of visibility checks not being truly required for SAOP inequalities like this one (when they appear in index filters). I'm not sure if that idea is too particular to SAOP inequalities to be interesting -- just a suggestion. -- Peter Geoghegan

Re: Use of additional index columns in rows filtering

2023-08-06 Thread Peter Geoghegan
On Sun, Aug 6, 2023 at 3:28 PM Peter Geoghegan wrote: > I decided to verify my understanding by checking what would happen > when I ran the OR-heavy tenk1 regression test query against a > combination of your patch, and v7 of the OR-to-SAOP transformation > patch. (To be clear, this i

Re: Use of additional index columns in rows filtering

2023-08-06 Thread Peter Geoghegan
On Sun, Aug 6, 2023 at 1:13 PM Peter Geoghegan wrote: > Since you're not relying on the nbtree work at all here, really (just > on the transformation process itself), the strategic risk that this > adds to your project isn't too great. It's not like this ties the > s

Re: POC, WIP: OR-clause support for indexes

2023-08-06 Thread Peter Geoghegan
On Sat, Aug 5, 2023 at 7:01 PM Peter Geoghegan wrote: > Of course this immediately makes me wonder: shouldn't your patch be > able to perform an additional transformation here? You know, by > transforming "a.x = 42 OR a.x = 44" into "a IN (42, 44)"? Although

Re: Use of additional index columns in rows filtering

2023-08-06 Thread Peter Geoghegan
lly penalize the paths that your patch generates -- so cases like the BitmapOr case should do the right thing by seeing the SAOP path as cheaper during planning. But is that approach really ideal for your patch? I doubt it. Why wouldn't we want to give the index paths with index filters credit for being cheaper in almost all cases? That's why this doesn't feel like a costing issue to me (it's more of a "just don't do that" issue, I think). Your patch seems too important to nerf like this, even if it's convenient in some ways. -- Peter Geoghegan

Re: POC, WIP: OR-clause support for indexes

2023-08-05 Thread Peter Geoghegan
e when and where the planner modifies state using RR is far more useful than it would be with regular GDB. Once I record a query, I find that I can "map out" what happens in the planner relatively easily. -- Peter Geoghegan

Re: Use of additional index columns in rows filtering

2023-08-04 Thread Peter Geoghegan
possible at all right now, that become possible, many of which don't even have SAOPs/ORs. Of course it won't ever be possible to create a transformation that doesn't ultimately flatten everything into MDAM style "single value" DNF predicates, which have to use simple B-Tree opclass operators -- obviously there are fundamental limits to it. So even in a perfect world, with every possible MDAM-ish transformation implemented, we'll still have a significant need for your patch. -- Peter Geoghegan

Re: Use of additional index columns in rows filtering

2023-08-04 Thread Peter Geoghegan
me issue. But I don't think it's the same issue in a deeper sense, since this is one case where you simply don't have to accept any risk. We really should be able to just not ever do this, for a limited though important subset of cases involving ORs + indexable operators. -- Peter Geoghegan

Re: Use of additional index columns in rows filtering

2023-08-03 Thread Peter Geoghegan
is most definitely not okay for "index filters" to displace equivalent "access predicate index quals", but it's also not really okay to allow them to displace equivalent "index filter predicate index quals" -- the latter case is less bad, but AFAICT they both basically aren't acceptable "substitutions".) -- Peter Geoghegan

Re: Use of additional index columns in rows filtering

2023-08-03 Thread Peter Geoghegan
ely small changes in one particular "thousand" grouping, with potentially outsized impact. And, this can happen very suddenly, because it's so "local". Making this plan perform robustly just doesn't seem to be one of the things that statistics can be expected to help us with very much. -- Peter Geoghegan

Re: Use of additional index columns in rows filtering

2023-08-03 Thread Peter Geoghegan
t; I feel a bit like the rubber duck from [1], but I'm OK with that ;-) Not from my point of view. Besides, even when somebody says that they just don't understand what I'm saying at all (which wasn't ever fully the case here), that is often useful feedback in itself. -- Peter Geoghegan

Re: Use of additional index columns in rows filtering

2023-08-03 Thread Peter Geoghegan
laced. The certainty about the upside is just as important as the uncertainty about the downside. The huge asymmetry matters, and is fairly atypical. If, somehow, there was less certainty about the possible upside, then my argument wouldn't really work. -- Peter Geoghegan

Re: Use of additional index columns in rows filtering

2023-08-03 Thread Peter Geoghegan
orth, I actually think that there is a good chance that I'll end up relying on what you've done here to make certain things I want to do with the SOAP patch okay. It would be rather convenient to be able to handle some of the SAOP safety issues without needing any table filters (just index filters), in some corner cases. I think that what you're doing here makes a lot of sense. FWIW, I am already personally invested in the success of your patch. -- Peter Geoghegan

Re: Use of additional index columns in rows filtering

2023-08-03 Thread Peter Geoghegan
to lose. Whereas at least the plan from the patch really is cheaper than the BitmapOr plan (the master branch plan from create_index.out) on average. -- Peter Geoghegan

Re: Use of additional index columns in rows filtering

2023-08-02 Thread Peter Geoghegan
On Wed, Aug 2, 2023 at 6:32 PM Peter Geoghegan wrote: > I don't dispute the fact that this can only happen when the planner > believes (with good reason) that the expected cost will be lower. But > I maintain that there is a novel risk to be concerned about, which is > meaningfu

Re: Use of additional index columns in rows filtering

2023-08-02 Thread Peter Geoghegan
said is not particularly helpful. At least not on its own. You quite naturally won't want to tie the fate of this patch to my SAOP patch, which is significantly more complicated. I do think that my concern about this being a novel risk needs to be carefully considered. Maybe it's possible to address my concern outside of the context of my own SAOP patch. That would definitely be preferable all around. But "access predicates versus filter predicates" seems important and relevant, either way. -- Peter Geoghegan

Re: POC, WIP: OR-clause support for indexes

2023-08-02 Thread Peter Geoghegan
of my own patch as far as I can, to see what breaks, and why it breaks. My intuition is that this will help me with my own patch by forcing me to gain a truly rigorous understanding of the problem. My suggestion about generalizing your approach to cover RowCompareExpr cases is what I would do, if I were you, and this was my patch. That's almost exactly what I'm doing with my own patch already, in fact. -- Peter Geoghegan

Re: Use of additional index columns in rows filtering

2023-08-01 Thread Peter Geoghegan
On Mon, Jul 24, 2023 at 11:59 AM Peter Geoghegan wrote: > > That might be true but I'm not sure what to do about that unless we > > incorporate some "robustness" measure into the costing. If every > > measure we have says one plan is better, don't we have

Re: POC, WIP: OR-clause support for indexes

2023-08-01 Thread Peter Geoghegan
aming to me. (Of course we'll still need to model skipping stuff in the planner -- just not so that we can decide between two index paths that are essentially identical, that should just be one index path.) [1] https://postgr.es/m/CAH2-Wz=ksvn_sjcnd1+bt-wtifra5ok48adynq3pkkhxgmq...@mail.gmail.com -- Peter Geoghegan

Re: POC, WIP: OR-clause support for indexes

2023-08-01 Thread Peter Geoghegan
s seem necessary when generating index paths in the optimizer. Other things seem necessary during preprocessing, in the nbtree code, at the start of the index scan. Unfortunately, it's not obvious to me where the right place is to deal with each aspect of setting up multi-column SAOP index quals. My mental model is very incomplete. -- Peter Geoghegan

Re: Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans, skip scan

2023-07-31 Thread Peter Geoghegan
the planner can initialize some preprocessing early. I think that we require a "two way conversation" between the planner and the index AM. -- Peter Geoghegan

Re: Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans, skip scan

2023-07-31 Thread Peter Geoghegan
ge after _bt_search as the previous > scan ended, we register that. I can see one advantage that block nested loop join would retain here: it does block-based accesses on both sides of the join. Since it "looks ahead" on both sides of the join, more repeat accesses are likely to be avoided. Not too sure how much that matters in practice, though. -- Peter Geoghegan

Re: POC, WIP: OR-clause support for indexes

2023-07-29 Thread Peter Geoghegan
order effects. We want to be open to useful second order effects that make query execution much faster in some specific context, while avoiding harmful second order effects. Intuitively, I think that it should be possible to do this with the transformations performed by your patch. In other words, "helpful serendipity" is an important advantage, while "harmful anti-serendipity" is what we really want to avoid. Ideally by making the harmful cases impossible "by construction". -- Peter Geoghegan

Re: Eager page freeze criteria clarification

2023-07-28 Thread Peter Geoghegan
ll be an important condition for triggering freezing in practice. Your question about 2 seems equivalent to "why not just always freeze?". I don't think that that's a bad question -- quite the opposite. Even trying to give an answer to this question would amount to getting involved in n

Re: Eager page freeze criteria clarification

2023-07-28 Thread Peter Geoghegan
oo much. As I said, I'm no longer interested in working on VACUUM (though I do hope that Melanie or somebody else picks up where I left off). I have nothing to say about any new work in this area. If you want me to do something in the scope of the work on 16, as a release management task, please be clear about what that is. -- Peter Geoghegan

Re: Eager page freeze criteria clarification

2023-07-28 Thread Peter Geoghegan
ing 17, though. I made a decision to stop working on VACUUM, though, so I'm afraid I won't be able to offer much help with any of this. (Happy to give more background information, though.) -- Peter Geoghegan

Re: Eager page freeze criteria clarification

2023-07-28 Thread Peter Geoghegan
docs in this area. My latest draft is here: https://postgr.es/m/CAH2-Wz=UUJz+MMb1AxFzz-HDA=1t1fx_kmrdovopzxkpa-t...@mail.gmail.com https://www.postgresql.org/message-id/attachment/146830/routine-vacuuming.html I've been meaning to get back to that, but other commitments have kept me from it. I'd welcome your involvement with that effort. -- Peter Geoghegan

Re: Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans, skip scan

2023-07-27 Thread Peter Geoghegan
ittle to say about B-Tree executor stuff -- it's almost all just describing the preprocessing/transformation process. It seems as if optimizations like the one from my patch were considered too obvious to talk about and/or out of scope by the authors. Thinking about the MDAM paper like that was what made everything fall into place for me. Remember, "missing key predicates" isn't all that special. -- Peter Geoghegan

Re: Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans, skip scan

2023-07-26 Thread Peter Geoghegan
else (index path clause safety issues notwithstanding). -- Peter Geoghegan

Re: Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans, skip scan

2023-07-26 Thread Peter Geoghegan
value predicates -- it's still just cycling through each distinct set of array keys in the patch. What you describe is a problem in theory, but I doubt that it's a problem in practice. You don't actually have to materialize the predicates up-front, or at all. Plus you can skip over them using the next index tuple. So skipping works both ways. -- Peter Geoghegan

Re: POC, WIP: OR-clause support for indexes

2023-07-25 Thread Peter Geoghegan
svn_sjcnd1+bt-wtifra5ok48adynq3pkkhxgmq...@mail.gmail.com Currently, the optimizer doesn't recognize multi-column indexes with SAOPs on every column as having a valid sort order, except on the first column. It seems possible that that has consequences for your patch. (I'm really only guessing, though; don't trust anything that I say about the optimizer too much.) -- Peter Geoghegan

Re: Show various offset arrays for heap WAL records

2023-07-25 Thread Peter Geoghegan
On Mon, Jul 10, 2023 at 10:29 PM Peter Geoghegan wrote: > > Let's add a src/backend/access/rmgrdesc/README file. We don't currently > > have any explanation anywhere why the rmgr desc functions are in a > > separate directory. The README would be a good place to expla

Optimizing nbtree ScalarArrayOp execution, allowing multi-column ordered scans, skip scan

2023-07-24 Thread Peter Geoghegan
lse. It really isn't that special. Right now I'm focussed on the big picture, in any case. I want to emphasize the very general nature of these techniques. Although I'm focussed on SOAPs in the short term, many queries that don't make use of SAOPs should ultimately see simil

Re: Use of additional index columns in rows filtering

2023-07-24 Thread Peter Geoghegan
ut cannot be an Index > Cond for one reason or another. I think that you're probably right about it being overly restrictive -- that was just a starting point for discussion. Perhaps there is an identifiable class of clauses that can benefit, but don't have the downside that I'm concerned about. -- Peter Geoghegan

Re: Use of additional index columns in rows filtering

2023-07-24 Thread Peter Geoghegan
etes" a more robust path in the sense that I've described. This is obviously true because there can't possibly be index quals/scan keys for non-key columns within the index AM. -- Peter Geoghegan

Re: Use of additional index columns in rows filtering

2023-07-23 Thread Peter Geoghegan
e both advantages, together. (To be clear, I'm certainly not suggesting that the example argues against what you want to do here -- it's just an example that jumped out at me.) Perhaps this example will make my confusion about the boundaries between each of our patches a bit more understandable. I was confused -- and I still am. I look forward to being less confused at some point in the future. -- Peter Geoghegan

Re: Use of additional index columns in rows filtering

2023-07-21 Thread Peter Geoghegan
't care about the index order, there may be a lot of value in making the nbtree code understand that this is an ordered index scan. That's what enables skipping, in all its forms (skipping individual comparisons, skipping whole subsections of the index, etc). I'm not saying that this is 100% problem free. But it seems like a promising high level direction. > In a way, focusing on the worst case does that by assuming the worst > combination - which is fine, although it may choose the slower (but > safer) approach in some cases. I don't think that it has to be slower on average (even by a tiny bit). It might just end up being slightly faster on average, and way faster on occasion. -- Peter Geoghegan

Re: Use of additional index columns in rows filtering

2023-07-20 Thread Peter Geoghegan
es() admits the estimates are > somewhat unreliable, so we can't expect to make perfect decisions. The attribute value independence assumption is wishful thinking, in no small part -- it's quite surprising that it works as well as it does, really. -- Peter Geoghegan

Re: Use of additional index columns in rows filtering

2023-07-19 Thread Peter Geoghegan
t you're discussing may relate to what I need to do with my patch. Right now my patch assumes that making SAOP clauses into proper index quals (that usually preserve index ordering) is an unalloyed good (when safe!). This assumption is approximately true on average, as far as I can tell. But it's probably quite untrue in various specific cases, that somebody is bound to care about. -- Peter Geoghegan

Re: Show various offset arrays for heap WAL records

2023-07-10 Thread Peter Geoghegan
ry. The README would be a good place to explain that, > and to have the formatting guidelines. See attached. I agree that it's better this way, though. -- Peter Geoghegan

Re: POC, WIP: OR-clause support for indexes

2023-06-27 Thread Peter Geoghegan
to gamble (in the planner, or at execution time). We're just keeping our options open in more cases. (My thinking on these topics was influenced by Goetz Graefe -- "choice is confusion" [2]). [1] https://www.postgresql.org/message-id/flat/1397.1486598083%40sss.pgh.pa.us#310f974a8dc

Re: ReadRecentBuffer() doesn't scale well

2023-06-27 Thread Peter Geoghegan
On Mon, Jun 26, 2023 at 11:27 PM Andres Freund wrote: > On 2023-06-26 21:53:12 -0700, Peter Geoghegan wrote: > > It should be safe to allow searchers to see a version of the root page > > that is out of date. The Lehman & Yao design is very permissive about > > these

Re: ReadRecentBuffer() doesn't scale well

2023-06-26 Thread Peter Geoghegan
ot;between levels". The root page isn't really special, except in the obvious way. We can even have two roots at the same time (the true root, and the fast root). -- Peter Geoghegan

Re: ReadRecentBuffer() doesn't scale well

2023-06-26 Thread Peter Geoghegan
g with the page LSN? -- Peter Geoghegan

Re: ReadRecentBuffer() doesn't scale well

2023-06-26 Thread Peter Geoghegan
the universe than the use case I was focusing on at > the time. They're not just the hottest. They're also among the least likely to change from one moment to the next. (If that ever failed to hold then it wouldn't take long for the index to become grotesquely tall.) -- Peter Geoghegan

Re: POC, WIP: OR-clause support for indexes

2023-06-25 Thread Peter Geoghegan
quot; to get the planner to use all of the quals from the query as index quals. It thinks (correctly) that the query plan is very inefficient. That happens to match reality right now, but the underlying reality could change significantly. Something to think about. -- Peter Geoghegan saop_patch_test.sql Description: Binary data

Re: Adding further hardening to nbtree page deletion

2023-06-21 Thread Peter Geoghegan
On Tue, Jun 20, 2023 at 11:13 PM Peter Geoghegan wrote: > FWIW, I'm almost certain that I'll completely run out of ERRORs to > demote to LOGs before too long. In fact, this might very well be the > last ERROR that I ever have to demote to a LOG to harden nbtree > VACUUM.

Re: Assert while autovacuum was executing

2023-06-20 Thread Peter Geoghegan
t. Did you know that ginInsertCleanup() is the only code that uses heavyweight page locks these days? Though only on the index metapage! Isn't this the kind of thing that VACUUM's relation level lock is supposed to take care of? -- Peter Geoghegan

Re: Adding further hardening to nbtree page deletion

2023-06-20 Thread Peter Geoghegan
he individual cases that I've addressed come up very infrequently in practice. -- Peter Geoghegan

Re: Adding further hardening to nbtree page deletion

2023-06-20 Thread Peter Geoghegan
On Fri, Jun 16, 2023 at 2:15 PM Peter Geoghegan wrote: > Attached patch adds additional hardening to nbtree page deletion. It > makes nbtree VACUUM tolerate a certain sort of cross-page > inconsistencies in the structure of an index (corruption). VACUUM can > press on, avoiding

Re: Optimizing "boundary cases" during backward scan B-Tree index descents

2023-06-20 Thread Peter Geoghegan
On Mon, Jun 19, 2023 at 4:28 PM Peter Geoghegan wrote: > We still fall short when it comes to handling boundary cases optimally > during backwards scans. This is at least true for a subset of > backwards scans that request "goback=true" processing inside > _bt_first. A

Optimizing "boundary cases" during backward scan B-Tree index descents

2023-06-19 Thread Peter Geoghegan
-- the insertion scankey code doesn't need to change at all. The real behavioral change takes place in _bt_first, the higher level calling code. It has been taught to set its insertion/initial positioning scankey's pivotsearch/goback field to "true" in the patch. Before now, this option was exclusively during VACUUM, for page deletion. It turns out that so-called "pivotsearch" behavior is far more general than currently supposed.) Thoughts? -- Peter Geoghegan v1-0001-Add-nbtree-goback-boundary-case-optimization.patch Description: Binary data

Re: Assert while autovacuum was executing

2023-06-17 Thread Peter Geoghegan
h the rule enforced by the assertion. -- Peter Geoghegan

Adding further hardening to nbtree page deletion

2023-06-16 Thread Peter Geoghegan
a robust approach to page deletion (after 2014 commit efada2b8e9). -- Peter Geoghegan v1-0001-nbtree-VACUUM-cope-with-topparent-inconsistencies.patch Description: Binary data

Re: Cleaning up nbtree after logical decoding on standby work

2023-06-10 Thread Peter Geoghegan
On Fri, Jun 9, 2023 at 12:23 PM Peter Geoghegan wrote: > > I'm not sure there is that concensus (for me half the changes shouldn't be > > done, the rest should be in 17), but in the end it doesn't matter that much. I pushed this just now. I have also closed out the

Re: Cleaning up nbtree after logical decoding on standby work

2023-06-09 Thread Peter Geoghegan
b313e4 hardly made any changes at all to GiST, even though the relevant parts of GiST are heavily based on nbtree. Did you just forget to plaster similar heaprel arguments all over GiST and SP-GiST? I'm really disappointed that you're still pushing back here, even after I got a +1 on backpatching from Heikki. This should have been straightforward. -- Peter Geoghegan

Re: Cleaning up nbtree after logical decoding on standby work

2023-06-09 Thread Peter Geoghegan
> > SortCoordinate coordinate, > > > > int sortopt); > > I think we should continue to provide the table here, even if we don't need it > today. I don't see why, but okay. I'll do it that way. -- Peter Geoghegan

Re: index prefetching

2023-06-09 Thread Peter Geoghegan
basis for prefetching. I *really* hate any kind of sharp discontinuity, where a minor change in an input (e.g., from minor amounts of heap fragmentation) has outsized impact on an output (e.g., buffers pinned). Interactions like that tend to be really pernicious -- they lead to bad performance that goes unnoticed and unfixed because the problem effectively camouflages itself. It may even be easier to make the conservative (perhaps paranoid) assumption that weird nasty interactions will cause harm somewhere down the line...why take a chance? I might end up prototyping this myself. I may have to put my money where my mouth is. :-) -- Peter Geoghegan

Re: index prefetching

2023-06-08 Thread Peter Geoghegan
On Thu, Jun 8, 2023 at 4:38 PM Peter Geoghegan wrote: > This is conceptually a "mini bitmap index scan", though one that takes > place "inside" a plain index scan, as it processes one particular leaf > page. That's the kind of design that "plain index sca

Re: index prefetching

2023-06-08 Thread Peter Geoghegan
certainly for v0. Prefetching > >> across multiple leaf pages seems way more complex (particularly for the > >> cases using pairing heap), so let's leave this for the future. > Yeah, I'm not saying it's impossible, and imagined we might teach nbtree &

Re: index prefetching

2023-06-08 Thread Peter Geoghegan
4 31479.4 > 156898.4 1600.7 187.5 32064.5 > 10 50975.215978.7 1848.9 31587.1 > > This is a good illustration of a query where bitmapscan is terrible > (much worse than seqscan, in fact), and the patch is a massive > improvement over master (about an order of magnitude). > > Of course, if you only scan a couple rows, the benefits are much more > modest (say 40% for 100 rows, which is still significant). Nice! And, it'll be nice to be able to use the kill_prior_tuple optimization in many more cases (possible by teaching the optimizer to favor index scans over bitmap index scans more often). -- Peter Geoghegan

Re: Cleaning up nbtree after logical decoding on standby work

2023-06-08 Thread Peter Geoghegan
as > small as possible. Attached is v4, which goes back to using "heaprel" in new-to-16 code. As a result, it is slightly smaller than v3. My new plan is to commit this tomorrow, since the clear consensus is that we should go ahead with this for 16. -- Peter Geoghegan v4-0001-nbtree-Allocate-new-pages-in-separate-function.patch Description: Binary data

Re: Cleaning up nbtree after logical decoding on standby work

2023-06-07 Thread Peter Geoghegan
re pretty much written on auto-pilot. I had no intention of making a fuss about it, but then I never expected this push back. -- Peter Geoghegan

Re: Let's make PostgreSQL multi-threaded

2023-06-05 Thread Peter Geoghegan
suggested that processes have significant robustness advantages. Maybe they do, but it's hard to say either way because these benefits only apply "when the impossible happens". In any given case it's reasonable to wonder if the user was protected by our multi-process architecture, or protected by dumb luck. Could even be both. -- Peter Geoghegan

Re: Let's make PostgreSQL multi-threaded

2023-06-05 Thread Peter Geoghegan
RITE_ACTIVITY() contains its own START_CRIT_SECTION(), despite not being involved in WAL logging. And so critical sections could indeed be described as something that we use whenever shared memory cannot be left in an inconsistent state (which often coincides with WAL logging, but need not). -- Peter Geoghegan

Re: Cleaning up nbtree after logical decoding on standby work

2023-06-05 Thread Peter Geoghegan
that _bt_allocbuf() is the one and only choke point where new pages/buffers can be allocated by nbtree, and the only possible source of recovery conflicts during REDO besides opportunistic deletion record conflicts -- so it really isn't strange for _bt_search callers to be thinking about whether

Re: Avoiding another needless ERROR during nbtree page deletion

2023-06-01 Thread Peter Geoghegan
particular issue, the error is "right sibling's left-link doesn't match". Per: https://stackoverflow.com/questions/49307292/error-in-postgresql-right-siblings-left-link-doesnt-match-block-5-links-to-8 -- Peter Geoghegan

Re: benchmark results comparing versions 15.2 and 16

2023-05-31 Thread Peter Geoghegan
=log; set trace_sort=on; $QUERY". You'll see lots of LOG messages with specific information about the use of abbreviated keys and the progress of each sort. Thanks -- Peter Geoghegan

Re: abi-compliance-checker

2023-05-29 Thread Peter Geoghegan
On Sun, May 28, 2023 at 9:34 AM Peter Geoghegan wrote: > I'll try to come up with a standard abi-compliance-checker Postgres > workflow once I'm back from pgCon. Ideally, we'd be able to produce reports that cover an entire stable release branch at once, including det

Re: benchmark results comparing versions 15.2 and 16

2023-05-29 Thread Peter Geoghegan
umns at that time (this was with glibc). Low cardinality inputs were more like 2.5x. I believe that ICU is faster than glibc in general -- even with TRUST_STRXFRM enabled. But the TRUST_STRXFRM thing is bound to be the most important factor here, by far. -- Peter Geoghegan

Re: abi-compliance-checker

2023-05-28 Thread Peter Geoghegan
guesswork out of ABI compatibility in stable releases, without any real downside, which is encouraging. I have spent very little time on this, so it's quite possible that some detail or other was overlooked. -- Peter Geoghegan

abi-compliance-checker

2023-05-27 Thread Peter Geoghegan
theoretical/actual x86_64 ABI breaks in each point release. I'd appreciate having greater visibility into these issues. [1] https://github.com/lvc/abi-dumper [2] https://manpages.debian.org/unstable/abi-dumper/abi-dumper.1.en.html -- Peter Geoghegan Title: libTest: X to Y compatibility

Re: Cleaning up nbtree after logical decoding on standby work

2023-05-26 Thread Peter Geoghegan
313e4 went in: https://postgr.es/m/CAH2-Wz=jgryxwm74g1khst0znpunhezyjnvsjno2t3jswtb...@mail.gmail.com -- Peter Geoghegan

Re: Cleaning up nbtree after logical decoding on standby work

2023-05-26 Thread Peter Geoghegan
ge() during its cluster tuplesort, no matter what. This has nothing to do with any underlying implementation detail from nbtree, or from any other index AM. -- Peter Geoghegan

Re: Cleaning up nbtree after logical decoding on standby work

2023-05-26 Thread Peter Geoghegan
On Fri, May 26, 2023 at 10:28 AM Peter Geoghegan wrote: > I've added several defensive assertions that make it hard to get the > details wrong. These will catch the issue much earlier than the main > "heapRel != NULL" assertion in _bt_allocbuf(). So, the rules are > rea

Re: Cleaning up nbtree after logical decoding on standby work

2023-05-26 Thread Peter Geoghegan
sive assertions that make it hard to get the details wrong. These will catch the issue much earlier than the main "heapRel != NULL" assertion in _bt_allocbuf(). So, the rules are reasonably straightforward and enforceable. -- Peter Geoghegan

Re: Cleaning up nbtree after logical decoding on standby work

2023-05-26 Thread Peter Geoghegan
te logging function. This structure seems like a clear improvement, since such logging is largely the point of having a separate _bt_allocbuf() function that deals with new page allocations and requires a valid heapRel in all cases. v2 also renames "heaprel" to "heapRel" in

Cleaning up nbtree after logical decoding on standby work

2023-05-25 Thread Peter Geoghegan
d page splits). It's possible that Bertand would have done it this way to begin with were it not for the admittedly pretty bad nbtree convention around P_NEW. It would be nice to get rid of P_NEW in the near future, too -- I gather that there was discussion of that in the context of recent work in t

Re: Avoiding another needless ERROR during nbtree page deletion

2023-05-25 Thread Peter Geoghegan
On Mon, May 22, 2023 at 10:59 AM Peter Geoghegan wrote: > Attached is v2, which does it that way. It also adjusts the approach > taken to release locks and pins when the left sibling validation check > fails. I pushed this just now, backpatching all the way. > Not including a rev

Re: Avoiding another needless ERROR during nbtree page deletion

2023-05-22 Thread Peter Geoghegan
On Mon, May 22, 2023 at 9:22 AM Peter Geoghegan wrote: > > This comment notes that this is similar to what we did with the left > > sibling, but there isn't really any mention at the left sibling code > > about avoiding hard ERRORs. Feels a bit backwards. Maybe move the >

Re: Avoiding another needless ERROR during nbtree page deletion

2023-05-22 Thread Peter Geoghegan
pens in the ShareLock case, so that we have everything covered. I'm unsure. > ERRCODE_NO_DATA doesn't look right. Let's just leave out the errcode. Agreed. -- Peter Geoghegan

<    1   2   3   4   5   6   7   8   9   10   >