the
underlying types sort), or seemed to discuss things that were better
discussed next to the relevant network_abbrev_convert() code.
Thoughts?
--
Peter Geoghegan
v3-0001-Add-sort-support-for-inet-cidr-opfamily.patch
Description: Binary data
implement A later."
I don't claim to have any real answers here. I don't claim to
understand how much of a problem this is.
[1] https://15721.courses.cs.cmu.edu/spring2016/papers/a16-graefe.pdf
[2] http://db.cs.berkeley.edu/papers/fntdb07-architecture.pdf -- See
"6.7 Standard Practice"
--
Peter Geoghegan
ity. Which seems to be why we
offer such a large variety of index access methods -- it's relatively
straight forward for Postgres to add niche index AMs, such as SP-GiST.
--
Peter Geoghegan
stgres. I agree that there is probably a middle way that more or
less has the advantages of both approaches. I don't really know what
that should look like, though.
[1]
https://www.microsoft.com/en-us/research/uploads/prod/2019/06/p700-antonopoulos.pdf
--
Peter Geoghegan
On Mon, Jul 29, 2019 at 12:39 PM Peter Geoghegan wrote:
> I think that indexes (or at least B-Tree indexes) will ideally almost
> always have tuples that are the latest versions with zheap. The
> exception is tuples whose ghost bit is set, whose visibility varies
> based on the M
On Fri, Aug 2, 2019 at 1:28 PM Julien Rouhaud wrote:
> I'm fine with it!
Pushed a version with similar wording just now.
Thanks!
--
Peter Geoghegan
d explicit type casts.
It's a contrib extension, so you have to "create extension amcheck" first.
--
Peter Geoghegan
always adjust the query to only verify
system indexes or TOAST indexes.
--
Peter Geoghegan
en to hear about how
much the tooling helps in the real world.
--
Peter Geoghegan
has some disadvantages that you might want
to avoid.)
Questions like this are better suited to the pgsql-general list.
--
Peter Geoghegan
mment to
_bt_insertonpg_in_posting() -- I don't think think that the code for
splitting a posting list in two is currently crash-safe.
How do you feel about officially calling this deduplication, not
compression? I think that it's a more accurate name for the technique.
--
Peter Geoghegan
v5-0001
Note that it is built on top of GNU units, which is itself highly extensible.
I'm not sure if this will be useful, since I am not an expert on
calendar systems.
--
Peter Geoghegan
hings easier to
> reason about.
Pushed.
Thanks!
--
Peter Geoghegan
ocks at just
the right points during execution.
If I had to guess, I'd guess that it had something to do with that. I
might be able to come up with a better explanation if I saw the diff.
--
Peter Geoghegan
e range 8000-. It's
just busy work.
--
Peter Geoghegan
is will ever be a real problem. Just try again.
> Wouldn't it be better to keep some room at the end of the allowed
> array? Or at least avoid suggesting ranges where there is less than
> 3-5 OIDs available consecutively.
Not in my view. There is value in having simple, predictable
whatever) far sooner than we'll run
out of single OIDs. Now we have to worry about doing a second
(actually a third) pass over the OIDs as a fallback when that happens.
And so on.
--
Peter Geoghegan
plesort-using operations less and less
these days -- the only remaining interesting bottleneck is probably in
code like index_form_tuple(), which is probably a good target for JIT.
In general, it's much harder to make tuplesort.c noticeably faster
than it used to be -- we've picked all the low-hanging fruit.
--
Peter Geoghegan
On Fri, Jul 19, 2019 at 12:32 PM Peter Geoghegan wrote:
> On Fri, Jul 19, 2019 at 10:53 AM Anastasia Lubennikova
> wrote:
> > Patch 0002 (must be applied on top of 0001) implements preserving of
> > correct TID order
> > inside posting list when inserting new tuples.
>
On Tue, Apr 30, 2019 at 5:58 PM Peter Geoghegan wrote:
> I will think about a simple fix, but after the upcoming point release.
> There is no hurry.
Attached draft patch uses RelationGetNumberOfBlocks() to size each of
the two Bloom filters that may be used by amcheck to perform
verifi
east because it probably requires cooperation from
underlying access methods.
--
Peter Geoghegan
are very likely to either be fixable with further
work on optimizing specific code, or a price that users will mostly be
happy to pay to get the benefits.
--
Peter Geoghegan
wavy, and I don't expect you to
have a definitive response. These are some high level concerns that I
have, that may or may not apply to what you're trying to do.
--
Peter Geoghegan
that I have it right, ISTM that UNDO/zheap/whatever should explicitly
own this restriction.
--
Peter Geoghegan
On Fri, Jul 19, 2019 at 7:24 PM Peter Geoghegan wrote:
> Hmm. So, the attached test case fails amcheck verification for me with
> the latest version of the patch:
Attached is a revised version of your v2 that fixes this issue -- I'll
call this v3. In general, my goal for the revision was t
.
This slightly simplifies the code, and also makes it marginally
faster. I'll add this to the 2019-09 CF.
--
Peter Geoghegan
0001-Overwrite-lastright-item-with-highkey-in-nbtsort.c.patch
Description: Binary data
, or when we step right during unique index
insertion.
This slightly simplifies the code. Our stack is inherently
approximate, because we might have to move right for a number of
reasons.
I'll add the patch to the 2019-09 CF.
--
Peter Geoghegan
0001-Remove-downlink-variable-from-nbtree-stack.patch
it;a=commit;h=c1afd175b5b2e5c44f6da34988342e00ecdfb518
It is a user facing feature, which increments the amcheck extension
version number.
Thanks
--
Peter Geoghegan
son to think that it's much different to what you've done with GiST?
--
Peter Geoghegan
ake a huge difference for most real world workloads.
OTOH, perhaps the risk is so low that we might as well target
backbranches.
How do you feel about it?
--
Peter Geoghegan
On Tue, Jul 23, 2019 at 6:22 PM Peter Geoghegan wrote:
> Attached is a revised version of your v2 that fixes this issue -- I'll
> call this v3.
Remember that index that I said was 5.5x smaller with the patch
applied, following retail insertions (a single big INSERT ... SELECT
...)? Well
es sense, there was no cases where locks would
> need to be carried forward.
I agree that this is unfortunate. Are you planning on working on it?
--
Peter Geoghegan
On Wed, Jul 24, 2019 at 3:06 PM Peter Geoghegan wrote:
> There seems to be a kind of "synergy" between the nbtsplitloc.c
> handling of pages that have lots of duplicates and posting list
> compression. It seems as if the former mechanism "sets up the bowling
> pins&q
he easy part.
--
Peter Geoghegan
On Thu, Jul 25, 2019 at 6:37 PM Bruce Momjian wrote:
> Attached patch applied, thanks.
Thanks Bruce,
--
Peter Geoghegan
On Fri, Feb 8, 2019 at 11:13 PM Edmund Horner wrote:
> I have some comments on the comments:
Seems reasonable to me.
Where are we on this? I'd like to get the patch committed soon.
--
Peter Geoghegan
; unset PGDATABASE unconditionally, let's try that and see how it
> goes.
It would be nice to get this fixed. Several people have been confused
by it at this point.
--
Peter Geoghegan
On Fri, Sep 27, 2019 at 2:39 PM Tom Lane wrote:
> I think I just forgot about this thread. Shall we change it in HEAD
> and see what happens? Maybe backpatch, but not till after 12.0 is out.
Please do.
--
Peter Geoghegan
On Mon, Sep 30, 2019 at 7:39 PM Peter Geoghegan wrote:
> I've found that my "regular pgbench, but with a low cardinality index
> on pgbench_accounts(abalance)" benchmark works best with the specific
> heuristics used in the patch, especially over many hours.
I
roach that applies
general rules (e.g. you can only do lock coupling left to right, or
bottom to top in nbtree). Anything less than that should be assumed to
be wrong on general principle.
--
Peter Geoghegan
rtCleanup() is concerned with pending list pages.
> BTW we found a small bug (wraparound) in similar GiST and B-tree
> implementations.
> Probably, it's there in GIN too.
Probably, but that's much less of a problem to me.
--
Peter Geoghegan
ead
> makes situation better not worse. That is it fixes one bug without
> introducing mode bugs. But I'm going to analyze more on this and
> document GIN concurrency better in the README. Probably, I'll spot
> more details.
Thanks.
--
Peter Geoghegan
eliminating that
factor (i.e. while setting fastupdate to off)?
Chen's example involved an INSERT that deadlocked against VACUUM --
not a SELECT. Is this just a coincidence?
--
Peter Geoghegan
fun:ServerLoop
fun:PostmasterMain
fun:main
}
""""""
I suspect that the recent commit 69f94108 is involved here, but I
haven't confirmed that explanation myself.
--
Peter Geoghegan
it.
The thread is here:
https://www.postgresql.org/message-id/flat/cah2-wzn3ee49gmxb7v1vj3-ac8fwn-fr8pfwqebhe8ryrxt...@mail.gmail.com
--
Peter Geoghegan
maller than the set that libpq
> reacts to --- we have added a ton of the latter without touching this
> list (much less the three or four other places that duplicate it).
> I wonder how problematic that is.
Only time will tell, I suspect.
--
Peter Geoghegan
maybe not, IMV.)
I ran pgindent on the patch, and made some small manual whitespace
adjustments, which is attached. There are no real changes, but some of
the formatting in the original version you posted was hard to read.
Please work off this for your next revision.
--
Peter Geoghegan
0001-btree_compression_pg12_v1.patch-with-pg_indent-run.patch
Description: Binary data
code for numeric abbreviated keys. We also have to worry
about pfree()'ing memory when USE_FLOAT8_BYVAL within
heapam_index_validate_scan(). How confident are we that there isn't
some place that leaks memory on !USE_FLOAT8_BYVAL builds because
somebody forgot to add a pfree() in an #ifdef block?
--
Peter Geoghegan
ate
on general principle. All good portable C codebases go to great
lengths to encapsulate platform differences, if necessary by adding a
compatibility layer. One of the worst things about the OpenSSL
codebase is that it makes writing portable code everybody's problem.
--
Peter Geoghegan
> a 32-bit platform --- more instructions, more memory consumed for
> > things like Datum arrays, all in a memory space that's not that big.
>
> I don't agree as well with the line of arguments to just remove 32b
> support.
Clearly you didn't read what I actually wrote, Michael.
--
Peter Geoghegan
On Sat, Nov 2, 2019 at 11:56 AM Peter Geoghegan wrote:
> On Wed, Sep 25, 2019 at 2:33 AM Dmitry Dolgov <9erthali...@gmail.com> wrote:
> > v27-0001-Index-skip-scan.patch
>
> Some random thoughts on this:
And now some more:
* I'm confused about this code in _bt_skip():
>
naming a little bit confusing.
Maybe rename _bt_skip() to something that is a little bit more
suggestive of its purpose.
* Suggest running pgindent on the patch.
[1]
https://wiki.postgresql.org/wiki/Index_Maintenance#Summarize_keyspace_of_a_B-Tree_index
--
Peter Geoghegan
but perhaps you have time for it.
--
Peter Geoghegan
ndexes will be in places
like vacuumlazy.c.
--
Peter Geoghegan
On Mon, Apr 22, 2019 at 9:35 AM Andres Freund wrote:
> On 2019-04-21 17:46:09 -0700, Peter Geoghegan wrote:
> > Andres has suggested that I work on teaching nbtree to accommodate
> > variable-width, logical table identifiers, such as those required for
> > indirect indexes,
ples are inserted that have
TIDs that are one byte wider, that go in the same posting list (as I
said, keeping the space accounting simple is particularly important
for nbtree). This even seems hard for GIN, which thinks of TIDs as an
array of fixed width ints in many contexts. Also, BRIN revmap pages
are also mostly just arrays of 6 byte item pointers, that rely on
simple pointer arithmetic to do random access.
--
Peter Geoghegan
uot;indirect" indexes can ever work in Postgres. I don't
know exactly what zedstore needs here, but maybe it can work well with
a less ambitious design for varwidth TIDs along the lines I've
sketched.
--
Peter Geoghegan
On Wed, Nov 13, 2019 at 11:33 AM Robert Haas wrote:
> On Tue, Nov 12, 2019 at 6:22 PM Peter Geoghegan wrote:
> > * Disabled deduplication in system catalog indexes by deeming it
> > generally unsafe.
>
> I (continue to) think that deduplication is a terrible name, because
olicy. (A catversion bump is
generally supposed to be done at the last minute, just as the patch is
committed. This avoids unnecessary conflicts against the master branch
over time, as a patch is developed.)
--
Peter Geoghegan
st put enough other words in there to make it clear e.g. duplicate
> value compression, or something of that sort.
Does anyone else want to weigh in on this? Anastasia?
I will go along with whatever the consensus is. I'm very close to the
problem we're trying to solve, which probably isn't helping me here.
--
Peter Geoghegan
On Fri, Nov 15, 2019 at 5:43 PM Mark Dilger wrote:
> On 11/13/19 11:51 AM, Peter Geoghegan wrote:
> > Can you suggest an alternative?
>
> Dupression
This suggestion makes me feel better about "deduplication".
--
Peter Geoghegan
slab.seas.harvard.edu/rum-conjecture/
--
Peter Geoghegan
On Wed, Sep 11, 2019 at 2:04 PM Peter Geoghegan wrote:
> > I haven't measured how these changes affect WAL size yet.
> > Do you have any suggestions on how to automate testing of new WAL records?
> > Is there any suitable place in regression tests?
>
> I don't know about
se the
non-reserved/development OID ranges directly. For example, a committer
may prefer to use an OID that is close to the OIDs already used for a
set of related objects, if the related objects are already in a stable
release. (I'm not sure that it's really worth doing that, but that's
what the policy is.)
--
Peter Geoghegan
On Fri, Nov 22, 2019 at 9:30 AM Ranier Vilela wrote:
>
> Hi,
> Typo mystake?
> Pointer var initilialized with boolean.
This was already fixed by commit 0cafdd03a850265006c0ada1b0bf4f83e087a409.
--
Peter Geoghegan
On Tue, Nov 12, 2019 at 3:21 PM Peter Geoghegan wrote:
> * Decided to go back to turning deduplication on by default with
> non-unique indexes, and off by default using unique indexes.
>
> The unique index stuff was regressed enough with INSERT-heavy
> workloads that I was put
On Wed, Nov 20, 2019 at 1:43 PM Peter Geoghegan wrote:
> My understanding is that we can trust RecentGlobalXmin to be something
> useful and current during recovery, in general, so the selfuncs.c
> index-only scan (which uses SnapshotNonVacuumable + RecentGlobalXmin)
> can be trusted
with commensurate reductions in query
latency) more recently. This was with a simple gaussian distribution
for pgbench_accounts.aid, and a non-unique index with deduplication
enabled on pgbench_accounts.abalance. (The patch helps control the
size of both indexes, especially the extra non-unique one.)
[1]
https://postgr.es/m/CAH2-WzkXHhjhmUYfVvu6afbojU97MST8RUT1U=hld2w-gc5...@mail.gmail.com
--
Peter Geoghegan
On Tue, Dec 17, 2019 at 5:18 PM Bruce Momjian wrote:
> On Tue, Dec 17, 2019 at 03:30:33PM -0800, Peter Geoghegan wrote:
> > With many real world unique indexes, the true reason behind most or
> > all B-Tree page splits is "version churn". I view these page splits as
On Fri, Nov 1, 2019 at 1:19 PM Robert Haas wrote:
> On Fri, Nov 1, 2019 at 3:15 PM Peter Geoghegan wrote:
> > I don't think that those two things are equivalent at all. There may
> > even be workloads that will benefit when run on 32-bit hardware.
> > Having to palloc() an
FS is rather immature.
How did you invoke pg_upgrade? Did you use the --link (hard link) option?
--
Peter Geoghegan
s will throw an error with an !indisvalid
index.)
--
Peter Geoghegan
it more on that angle.
That's hard for me to test right now, but offhand this general
approach looks good to me. I am pretty sure it's portable.
--
Peter Geoghegan
k home from my trip to Europe -- it
uses an SD card, which is pretty slow.
> I'm not saying that tuplesort has gotten worse or anything. Just that
> there's been too much development without adding tests.
I agree.
--
Peter Geoghegan
gt; I'd like you to take a look on the README patch first.
Thank you for working on this.
I am flying back to the USA today, and will try to take a look at what
you came up with on the way. I will definitely have some feedback in
the next few days.
--
Peter Geoghegan
On Tue, Oct 29, 2019 at 1:09 PM Andres Freund wrote:
> Comments?
I think that this is a good idea. I see no downside.
--
Peter Geoghegan
ow the
key space works. The nbtree work for v12 greatly benefitted from
defining comparisons in a way that didn't really change how nbtree
worked, while at the same time minimizing I/O and making nbtree
faithful to Lehman & Yao's original design. It isn't obvious how
valuable it is to really carefully define how invariants and key
comparisons work, but it seems possible to solve a lot of problems
that way.
--
Peter Geoghegan
one field from that record, for no
good reason. I backpatched a bugfix to the output format for nbtree
page splits a few weeks ago, fixing that problem. I agree that we
should also backpatch this bugfix.
--
Peter Geoghegan
> nontrivial areas without any tests - do we have actually have any
> confidence that they work?
Everything that you're talking about has existed since v11 came out a
year ago, and most of it is a year or two older than that. So yeah,
I'm pretty confident that it works.
--
Peter Geoghegan
front.
> I don't know what that means.
Instead of starting new email threads for each issue, confine the
entire discussion to just one thread. This makes the discussion much
more manageable for everyone else. This is a high traffic mailing
list.
--
Peter Geoghegan
this specific esoteric function,
which is called when we've already decided to split the page (but only
sometimes). Sanitization needs to happen at some central choke point.
> Yes,two static tools, but reviewed by me.
I strongly suggest confining all of this to a single thread, and
stating your reasoning upfront.
--
Peter Geoghegan
tatic analysis tool?
--
Peter Geoghegan
ugh to justify keeping all the SIZEOF_DATUM crud around.
--
Peter Geoghegan
ckVacuumed (we also don't need to call
_bt_delitems_vacuum() one last time in the case where we don't have
anything to kill on the last block, just so the pin scan can happen --
it won't ever happen).
--
Peter Geoghegan
On Wed, Nov 20, 2019 at 6:07 PM Michael Paquier wrote:
> Yep, agreed. This looks like an oversight. Peter?
It's not an oversight. See the commit message of a6417078, and the
additions that were made to the RELEASE_CHANGES file.
--
Peter Geoghegan
m OID from this range, so this will probably be
very common soon.
--
Peter Geoghegan
On Mon, Sep 23, 2019 at 5:13 PM Peter Geoghegan wrote:
> I attach version 17.
I attach a patch that applies on top of v17. It adds support for
deduplication within unique indexes. Actually, this is a snippet of
code that appeared in my prototype from August 5 (we need very little
extra c
r, do we store both pieces of information together, and forbid
setting the storage parameter to on when it's known to be unsafe for
the underlying opclasses used by the index? I don't know.
I think that you can start working on this without knowing exactly how
we'll do those catalog lookups. What
or if we have a full row and memory is getting
* full.
- *
- * XXX using up maintenance_work_mem here is probably unreasonably
- * much, since vacuum might already be using that much.
*/
ISTM that the use of maintenance_work_mem wasn't given that much
thought originally.
--
Peter Geoghegan
page dump is described here:
https://wiki.postgresql.org/wiki/Getting_a_stack_trace_of_a_running_PostgreSQL_backend_on_Linux/BSD#contrib.2Fpageinspect_page_dump
If I had to guess, I'd guess that this was due to a generic storage problem.
--
Peter Geoghegan
On Thu, Oct 10, 2019 at 1:18 PM Peter Geoghegan wrote:
> You didn't say which PostgreSQL versions were involved, and if the
> database was ever upgraded using pg_upgrade. Those details could
> matter.
In case you weren't aware, contrib/amcheck should make detected and
diagnosing th
gative availMem sizes --
when huge allocations were first supported, tuplesort.c briefly used
"Size", which didn't work. Perhaps it had something to do with that.
--
Peter Geoghegan
exists at all, and the fact that the sidelinks still exist at all.
This smells like a problem with slightly inconsistent page images, as
opposed to a problem with how one particular atomic operation did
something. It's not actually surprising that this would be the first
place that you'd notice a generic issue, since many other things are
"more forgiving" in various ways.
--
Peter Geoghegan
e assignment of "opaque". The sequence of BufferGetPage()
> and PageGetSpecialPointer() is a very standard switch-our-attention-
> to-another-page locution in nbtree and other index AMs.
+1
--
Peter Geoghegan
hing, and recreate everything, since
your indexes should be considered corrupt anyway. (Also, I don't think
that it's that hard to get it right, so this will probably never
happen.)
--
Peter Geoghegan
On Thu, Dec 12, 2019 at 6:21 PM Peter Geoghegan wrote:
> Still waiting for some review of the first patch, to get it out of the
> way. Anastasia?
I plan to commit this first patch [1] in the next day or two, barring
any objections.
It's clear that the nbtree "pin scan" VACUUM
oss-type support function 4 entries for an
opclass, on the grounds that that isn't sensible for deduplication. Do
you think that that restriction is appropriate in general, given the
likelihood that this support function will be used in several other
areas?
Thanks
--
Peter Geoghegan
On Mon, Feb 24, 2020 at 4:54 PM Peter Geoghegan wrote:
> Attached is v34, which has this change. My plan is to commit something
> very close to this on Wednesday morning (barring any objections).
Pushed.
I'm going to delay committing the pageinspect patch until tomorrow,
since I haven't t
, of course). If I come up with a
similar example that returns 10 rows (i.e. that indexes the "thousand"
row instead), random_page_cost needs to be reduced to 1.1 to produce
an equivalent query plan crossover.
--
Peter Geoghegan
tup2)
> +{
v7 looks pretty close to being commitable, though I'll probably want
to update some comments that you haven't touched when you commit this.
I should probably wait until you've committed the patch to go do that.
I'm thinking of things like old comments in bt_downlink_check().
I will test the patch properly one more time when you produce a new
revision. I haven't really tested it since the last time.
--
Peter Geoghegan
801 - 900 of 3113 matches
Mail list logo
