On Tue, Jul 4, 2023 at 12:49 PM Masahiko Sawada
wrote:
>
> As you mentioned, the 1-byte value is embedded into 8 byte so 7 bytes
> are unused, but we use less memory since we use less slab contexts and
> save fragmentations.
Thanks for testing. This tree is sparse enough that most of the space
On Tue, Jun 27, 2023 at 5:20 PM Masahiko Sawada wrote:
>
> On Fri, Jun 23, 2023 at 6:54 PM John Naylor
> wrote:
> >
> >
> > I wrote:
> > > I cleaned up a few things and attached v34 so you can do that if you like.
> >
> > Of course, "clean" is a relative term. While making a small bit of
On Fri, Jun 23, 2023 at 6:54 PM John Naylor
wrote:
>
>
> I wrote:
> > I cleaned up a few things and attached v34 so you can do that if you like.
>
> Of course, "clean" is a relative term. While making a small bit of progress
> working in tidbitmap.c earlier this week, I thought it useful to
On Tue, Jun 13, 2023 at 12:47 PM Masahiko Sawada
wrote:
>
> On Tue, Jun 6, 2023 at 2:13 PM John Naylor
wrote:
> >
> > I'd need to make sure, but apparently just going from 6 non-empty
memory contexts to 3 (remember all values are embedded here) reduces memory
fragmentation significantly in this
On Tue, Jun 6, 2023 at 2:13 PM John Naylor wrote:
>
> On Mon, Jun 5, 2023 at 5:32 PM Masahiko Sawada wrote:
> >
> > > Sometime in the not-too-distant future, I will start a new thread
> > > focusing on bitmap heap scan, but for now, I just want to share some
> > > progress on making the radix
On Mon, Jun 5, 2023 at 5:32 PM Masahiko Sawada
wrote:
>
> > Sometime in the not-too-distant future, I will start a new thread
focusing on bitmap heap scan, but for now, I just want to share some
progress on making the radix tree usable not only for that, but hopefully a
wider range of
Hi,
On Tue, May 23, 2023 at 7:17 PM John Naylor
wrote:
>
> I wrote:
> > the current insert/delete paths are quite complex. Using bitmap heap scan
> > as a motivating use case, I hope to refocus complexity to where it's most
> > needed, and aggressively simplify where possible.
>
> Sometime in
I wrote:
> the current insert/delete paths are quite complex. Using bitmap heap scan
as a motivating use case, I hope to refocus complexity to where it's most
needed, and aggressively simplify where possible.
Sometime in the not-too-distant future, I will start a new thread focusing
on bitmap
On Fri, Apr 7, 2023 at 4:55 PM John Naylor
wrote:
> - Fixed-size PagetableEntry's are pretty large, but the tid compression
scheme used in this thread (in addition to being complex) is not a great
fit for tidbitmap because it makes it more difficult to track per-block
metadata (see also next
On Wed, Apr 19, 2023 at 4:02 PM John Naylor
wrote:
>
> On Mon, Apr 17, 2023 at 8:49 PM Masahiko Sawada wrote:
>
> > > - With lazy expansion and single-value leaves, the root of a radix tree
> > > can point to a single leaf. That might get rid of the need to track
> > > TBMStatus, since setting
On Mon, Apr 17, 2023 at 8:49 PM Masahiko Sawada
wrote:
> > - With lazy expansion and single-value leaves, the root of a radix tree
can point to a single leaf. That might get rid of the need to track
TBMStatus, since setting a single-leaf tree should be cheap.
> >
>
> Instead of introducing
On Fri, Apr 7, 2023 at 6:55 PM John Naylor wrote:
>
> On Thu, Feb 16, 2023 at 11:44 PM Andres Freund wrote:
> >
> > We really ought to replace the tid bitmap used for bitmap heap scans. The
> > hashtable we use is a pretty awful data structure for it. And that's not
> > filled in-order, for
On Thu, Feb 16, 2023 at 11:44 PM Andres Freund wrote:
>
> We really ought to replace the tid bitmap used for bitmap heap scans. The
> hashtable we use is a pretty awful data structure for it. And that's not
> filled in-order, for example.
I spent some time studying tidbitmap.c, and not only does
On Tue, Mar 21, 2023 at 2:41 PM John Naylor
wrote:
>
>
> On Mon, Mar 20, 2023 at 9:34 PM Masahiko Sawada wrote:
> >
> > On Mon, Mar 20, 2023 at 9:34 PM John Naylor
> > wrote:
> > > That's an interesting idea, and the analogous behavior to aset could be a
> > > good thing for readability and
On Mon, Mar 20, 2023 at 9:34 PM Masahiko Sawada
wrote:
>
> On Mon, Mar 20, 2023 at 9:34 PM John Naylor
> wrote:
> > That's an interesting idea, and the analogous behavior to aset could be
a good thing for readability and maintainability. Worth seeing if it's
workable.
>
> I've attached a quick
On Mon, Mar 20, 2023 at 9:34 PM John Naylor
wrote:
>
>
> On Mon, Mar 20, 2023 at 12:25 PM Masahiko Sawada
> wrote:
> >
> > On Fri, Mar 17, 2023 at 4:49 PM Masahiko Sawada
> > wrote:
> > >
> > > On Fri, Mar 17, 2023 at 4:03 PM John Naylor
> > > wrote:
> > > >
> > > > On Wed, Mar 15, 2023 at
On Mon, Mar 20, 2023 at 12:25 PM Masahiko Sawada
wrote:
>
> On Fri, Mar 17, 2023 at 4:49 PM Masahiko Sawada
wrote:
> >
> > On Fri, Mar 17, 2023 at 4:03 PM John Naylor
> > wrote:
> > >
> > > On Wed, Mar 15, 2023 at 9:32 AM Masahiko Sawada
wrote:
> > > >
> > > > On Tue, Mar 14, 2023 at 8:27 PM
On Fri, Mar 17, 2023 at 4:49 PM Masahiko Sawada wrote:
>
> On Fri, Mar 17, 2023 at 4:03 PM John Naylor
> wrote:
> >
> > On Wed, Mar 15, 2023 at 9:32 AM Masahiko Sawada
> > wrote:
> > >
> > > On Tue, Mar 14, 2023 at 8:27 PM John Naylor
> > > wrote:
> > > >
> > > > I wrote:
> > > >
> > > > > >
On Fri, Mar 17, 2023 at 4:03 PM John Naylor
wrote:
>
> On Wed, Mar 15, 2023 at 9:32 AM Masahiko Sawada wrote:
> >
> > On Tue, Mar 14, 2023 at 8:27 PM John Naylor
> > wrote:
> > >
> > > I wrote:
> > >
> > > > > > Since the block-level measurement is likely overestimating quite a
> > > > > >
On Wed, Mar 15, 2023 at 9:32 AM Masahiko Sawada
wrote:
>
> On Tue, Mar 14, 2023 at 8:27 PM John Naylor
> wrote:
> >
> > I wrote:
> >
> > > > > Since the block-level measurement is likely overestimating quite
a bit, I propose to simply reverse the order of the actions here,
effectively reporting
On Tue, Mar 14, 2023 at 8:27 PM John Naylor
wrote:
>
> I wrote:
>
> > > > Since the block-level measurement is likely overestimating quite a bit,
> > > > I propose to simply reverse the order of the actions here, effectively
> > > > reporting progress for the *last page* and not the current
I wrote:
> > > Since the block-level measurement is likely overestimating quite a
bit, I propose to simply reverse the order of the actions here, effectively
reporting progress for the *last page* and not the current one: First
update progress with the current memory usage, then add tids for this
On Mon, Mar 13, 2023 at 10:28 PM John Naylor
wrote:
>
> On Mon, Mar 13, 2023 at 8:41 AM Masahiko Sawada wrote:
> >
> > On Sun, Mar 12, 2023 at 12:54 AM John Naylor
> > wrote:
> > >
> > > On Fri, Mar 10, 2023 at 9:30 PM Masahiko Sawada
> > > wrote:
>
> > > > * Additional size classes. It's
On Mon, Mar 13, 2023 at 8:41 AM Masahiko Sawada
wrote:
>
> On Sun, Mar 12, 2023 at 12:54 AM John Naylor
> wrote:
> >
> > On Fri, Mar 10, 2023 at 9:30 PM Masahiko Sawada
wrote:
> > > * Additional size classes. It's important for an alternative of path
> > > compression as well as supporting our
On Sun, Mar 12, 2023 at 12:54 AM John Naylor
wrote:
>
> On Fri, Mar 10, 2023 at 9:30 PM Masahiko Sawada wrote:
> >
> > On Fri, Mar 10, 2023 at 3:42 PM John Naylor
> > wrote:
>
> > > I'd suggest sharing your todo list in the meanwhile, it'd be good to
> > > discuss what's worth doing and what
On Fri, Mar 10, 2023 at 9:30 PM Masahiko Sawada
wrote:
>
> On Fri, Mar 10, 2023 at 3:42 PM John Naylor
> wrote:
> > I'd suggest sharing your todo list in the meanwhile, it'd be good to
discuss what's worth doing and what is not.
>
> Apart from more rounds of reviews and tests, my todo items
On Fri, Mar 10, 2023 at 3:42 PM John Naylor
wrote:
>
> On Thu, Mar 9, 2023 at 1:51 PM Masahiko Sawada wrote:
>
> > I've attached the new version patches. I merged improvements and fixes
> > I did in the v29 patch.
>
> I haven't yet had a chance to look at those closely, since I've had to devote
On Thu, Mar 9, 2023 at 1:51 PM Masahiko Sawada
wrote:
> I've attached the new version patches. I merged improvements and fixes
> I did in the v29 patch.
I haven't yet had a chance to look at those closely, since I've had to
devote time to other commitments. I remember I wasn't particularly
On Tue, Mar 7, 2023 at 8:25 AM Masahiko Sawada
wrote:
> > 1. Make it optional to track chunk memory space by a template
parameter. It might be tiny compared to everything else that vacuum does.
That would allow other users to avoid that overhead.
> > 2. When context block usage exceeds the limit
On Tue, Mar 7, 2023 at 1:01 AM John Naylor wrote:
>
> On Mon, Mar 6, 2023 at 1:28 PM Masahiko Sawada wrote:
>
> > > Since the block-level measurement is likely overestimating quite a bit, I
> > > propose to simply reverse the order of the actions here, effectively
> > > reporting progress for
On Mon, Mar 6, 2023 at 1:28 PM Masahiko Sawada
wrote:
> > Since the block-level measurement is likely overestimating quite a bit,
I propose to simply reverse the order of the actions here, effectively
reporting progress for the *last page* and not the current one: First
update progress with the
On Fri, Mar 3, 2023 at 8:04 PM John Naylor wrote:
>
> On Wed, Mar 1, 2023 at 6:59 PM Masahiko Sawada wrote:
> >
> > On Wed, Mar 1, 2023 at 3:37 PM John Naylor
> > wrote:
> > >
> > > I think we're trying to solve the wrong problem here. I need to study
> > > this more, but it seems that code
On Wed, Mar 1, 2023 at 6:59 PM Masahiko Sawada
wrote:
>
> On Wed, Mar 1, 2023 at 3:37 PM John Naylor
wrote:
> >
> > I think we're trying to solve the wrong problem here. I need to study
this more, but it seems that code that needs to stay within a memory limit
only needs to track what's been
On Wed, Mar 1, 2023 at 3:37 PM John Naylor wrote:
>
> On Tue, Feb 28, 2023 at 10:09 PM Masahiko Sawada
> wrote:
> >
> > On Tue, Feb 28, 2023 at 10:20 PM Masahiko Sawada
> > wrote:
> > >
> > > On Tue, Feb 28, 2023 at 3:42 PM John Naylor
> > > wrote:
> > > >
> > > >
> > > > On Fri, Feb 24,
On Tue, Feb 28, 2023 at 10:09 PM Masahiko Sawada
wrote:
>
> On Tue, Feb 28, 2023 at 10:20 PM Masahiko Sawada
wrote:
> >
> > On Tue, Feb 28, 2023 at 3:42 PM John Naylor
> > wrote:
> > >
> > >
> > > On Fri, Feb 24, 2023 at 12:50 PM Masahiko Sawada <
sawada.m...@gmail.com> wrote:
> > > >
> > > >
On Tue, Feb 28, 2023 at 10:20 PM Masahiko Sawada wrote:
>
> On Tue, Feb 28, 2023 at 3:42 PM John Naylor
> wrote:
> >
> >
> > On Fri, Feb 24, 2023 at 12:50 PM Masahiko Sawada
> > wrote:
> > >
> > > On Wed, Feb 22, 2023 at 6:55 PM John Naylor
> > > wrote:
> > > >
> > > > That doesn't seem
On Tue, Feb 28, 2023 at 3:42 PM John Naylor
wrote:
>
>
> On Fri, Feb 24, 2023 at 12:50 PM Masahiko Sawada
> wrote:
> >
> > On Wed, Feb 22, 2023 at 6:55 PM John Naylor
> > wrote:
> > >
> > > That doesn't seem useful to me. If we've done enough testing to reassure
> > > us the new way always
On Fri, Feb 24, 2023 at 12:50 PM Masahiko Sawada
wrote:
>
> On Wed, Feb 22, 2023 at 6:55 PM John Naylor
> wrote:
> >
> > That doesn't seem useful to me. If we've done enough testing to
reassure us the new way always gives the same answer, the old way is not
needed at commit time. If there is any
On Fri, Feb 24, 2023 at 3:41 PM Masahiko Sawada
wrote:
>
> In v29 vacuum took twice as long (286 s vs. 573 s)?
Not sure what happened there, and clearly I was looking at the wrong number
:/
I scripted the test for reproducibility and ran it three times. Also
included some variations (attached):
On Thu, Feb 23, 2023 at 6:41 PM John Naylor
wrote:
>
> I ran a couple "in situ" tests on server hardware using UUID columns, since
> they are common in the real world and have bad correlation to heap order, so
> are a challenge for index vacuum.
Thank you for the test!
>
> === test 1, delete
On Wed, Feb 22, 2023 at 6:55 PM John Naylor
wrote:
>
>
> On Wed, Feb 22, 2023 at 3:29 PM Masahiko Sawada wrote:
> >
> > On Wed, Feb 22, 2023 at 4:35 PM John Naylor
> > wrote:
> > >
> > > I don't think any vacuum calls in regression tests would stress any of
> > > this code very much, so it's
I ran a couple "in situ" tests on server hardware using UUID columns, since
they are common in the real world and have bad correlation to heap
order, so are a challenge for index vacuum.
=== test 1, delete everything from a small table, with very small
maintenance_work_mem:
alter system set
On Wed, Feb 22, 2023 at 3:29 PM Masahiko Sawada
wrote:
>
> On Wed, Feb 22, 2023 at 4:35 PM John Naylor
> wrote:
> >
> > I don't think any vacuum calls in regression tests would stress any of
this code very much, so it's not worth carrying the old way forward. I was
thinking of only doing this
On Wed, Feb 22, 2023 at 4:35 PM John Naylor
wrote:
>
>
> On Wed, Feb 22, 2023 at 1:16 PM Masahiko Sawada wrote:
> >
> > On Mon, Feb 20, 2023 at 2:56 PM Masahiko Sawada
> > wrote:
> > >
> > > Yeah, I did a similar thing in an earlier version of tidstore patch.
>
> Okay, if you had checks
On Wed, Feb 22, 2023 at 1:16 PM Masahiko Sawada
wrote:
>
> On Mon, Feb 20, 2023 at 2:56 PM Masahiko Sawada
wrote:
> >
> > Yeah, I did a similar thing in an earlier version of tidstore patch.
Okay, if you had checks against the old array lookup in development, that
gives us better confidence.
>
On Mon, Feb 20, 2023 at 2:56 PM Masahiko Sawada wrote:
>
> On Thu, Feb 16, 2023 at 6:23 PM John Naylor
> wrote:
> >
> > On Thu, Feb 16, 2023 at 10:24 AM Masahiko Sawada
> > wrote:
> > >
> > > On Tue, Feb 14, 2023 at 8:24 PM John Naylor
> > > wrote:
> >
> > > > > I can think that something
On Thu, Feb 16, 2023 at 11:44 PM Andres Freund wrote:
>
> Hi,
>
> On 2023-02-16 16:22:56 +0700, John Naylor wrote:
> > On Thu, Feb 16, 2023 at 10:24 AM Masahiko Sawada
> > > Right. TidStore is implemented not only for heap, so loading
> > > out-of-order TIDs might be important in the future.
> >
Hi,
On 2023-02-16 16:22:56 +0700, John Naylor wrote:
> On Thu, Feb 16, 2023 at 10:24 AM Masahiko Sawada
> > Right. TidStore is implemented not only for heap, so loading
> > out-of-order TIDs might be important in the future.
>
> That's what I was probably thinking about some weeks ago, but I'm
On Thu, Feb 16, 2023 at 10:24 AM Masahiko Sawada
wrote:
>
> On Tue, Feb 14, 2023 at 8:24 PM John Naylor
> wrote:
> > > I can think that something like traversing a HOT chain could visit
> > > offsets out of order. But fortunately we prune such collected TIDs
> > > before heap vacuum in heap
On Tue, Feb 14, 2023 at 8:24 PM John Naylor
wrote:
>
> On Mon, Feb 13, 2023 at 2:51 PM Masahiko Sawada wrote:
> >
> > On Sat, Feb 11, 2023 at 2:33 PM John Naylor
> > wrote:
> > >
> > > I didn't get any closer to radix-tree regression,
> >
> > Me neither. It seems that in v26, inserting chunks
On Sat, Feb 11, 2023 at 2:33 PM John Naylor
wrote:
>
> I didn't get any closer to radix-tree regression,
Me neither. It seems that in v26, inserting chunks into node-32 is
slow but needs more analysis. I'll share if I found something
interesting.
> but I did find some inefficiencies in
I didn't get any closer to radix-tree regression, but I did find some
inefficiencies in tidstore_add_tids() that are worth talking about first,
addressed in a rough fashion in the attached .txt addendums that I can
clean up and incorporate later.
To start, I can reproduce the regression with this
On Fri, Feb 10, 2023 at 3:51 PM John Naylor
wrote:
>
>
> On Thu, Feb 9, 2023 at 2:08 PM Masahiko Sawada wrote:
> >
> > query: select * from bench_tidstore_load(0, 10 * 1000 * 1000)
> >
> > v15:
> > load_ms
> > -
> > 816
>
> How did you build the tid store and test on v15? I first
On Thu, Feb 9, 2023 at 2:08 PM Masahiko Sawada
wrote:
>
> query: select * from bench_tidstore_load(0, 10 * 1000 * 1000)
>
> v15:
> load_ms
> -
> 816
How did you build the tid store and test on v15? I first tried to
apply v15-0009-PoC-lazy-vacuum-integration.patch, which conflicts
On Thu, Feb 9, 2023 at 2:08 PM Masahiko Sawada
wrote:
> I think it's still important for lazy vacuum that an iteration over a
> TID store returns TIDs in ascending order, because otherwise a heap
> vacuum does random writes. That being said, we can have
> RT_ITERATE_NEXT() return key-value pairs
Hi,
On Tue, Feb 7, 2023 at 6:25 PM John Naylor wrote:
>
>
> On Tue, Jan 31, 2023 at 9:43 PM Masahiko Sawada wrote:
>
> > I've attached v24 patches. The locking support patch is separated
> > (0005 patch). Also I kept the updates for TidStore and the vacuum
> > integration from v23 separate.
>
>
On Mon, Jan 30, 2023 at 1:31 PM John Naylor
wrote:
>
> On Sun, Jan 29, 2023 at 9:50 PM Masahiko Sawada wrote:
> >
> > On Sat, Jan 28, 2023 at 8:33 PM John Naylor
> > wrote:
>
> > > The first implementation should be simple, easy to test/verify, easy to
> > > understand, and easy to replace. As
On Mon, Jan 30, 2023 at 1:08 PM Dilip Kumar wrote:
>
> On Thu, Jan 26, 2023 at 12:39 PM John Naylor
> wrote:
> >
> >
> > On Tue, Jan 24, 2023 at 1:17 PM Dilip Kumar wrote:
> > >
> > > On Mon, Jan 23, 2023 at 6:00 PM John Naylor
> > > wrote:
> > > >
> > > > Attached is a rebase to fix conflicts
On Sun, Jan 29, 2023 at 9:50 PM Masahiko Sawada
wrote:
>
> On Sat, Jan 28, 2023 at 8:33 PM John Naylor
> wrote:
> > The first implementation should be simple, easy to test/verify, easy to
understand, and easy to replace. As much as possible anyway.
>
> Yes, but if a concurrent writer waits for
On Thu, Jan 26, 2023 at 12:39 PM John Naylor
wrote:
>
>
> On Tue, Jan 24, 2023 at 1:17 PM Dilip Kumar wrote:
> >
> > On Mon, Jan 23, 2023 at 6:00 PM John Naylor
> > wrote:
> > >
> > > Attached is a rebase to fix conflicts from recent commits.
> >
> > I have reviewed v22-0022* patch and I have
On Sat, Jan 28, 2023 at 8:33 PM John Naylor
wrote:
>
> On Thu, Jan 26, 2023 at 3:33 PM Masahiko Sawada wrote:
> >
> > On Thu, Jan 26, 2023 at 3:54 PM John Naylor
> > wrote:
>
> > I think that we need to prevent concurrent updates (RT_SET() and
> > RT_DELETE()) during the iteration to get the
On Thu, Jan 26, 2023 at 3:33 PM Masahiko Sawada
wrote:
>
> On Thu, Jan 26, 2023 at 3:54 PM John Naylor
> wrote:
> I think that we need to prevent concurrent updates (RT_SET() and
> RT_DELETE()) during the iteration to get the consistent result through
> the whole iteration operation. Unlike
On Thu, Jan 26, 2023 at 3:54 PM John Naylor
wrote:
>
> On Wed, Jan 25, 2023 at 8:42 AM Masahiko Sawada wrote:
> >
> > On Mon, Jan 23, 2023 at 8:20 PM John Naylor
> > wrote:
> > >
> > > On Mon, Jan 16, 2023 at 3:18 PM Masahiko Sawada
> > > wrote:
> > > >
> > > > On Mon, Jan 16, 2023 at 2:02 PM
On Tue, Jan 24, 2023 at 1:17 PM Dilip Kumar wrote:
>
> On Mon, Jan 23, 2023 at 6:00 PM John Naylor
> wrote:
> >
> > Attached is a rebase to fix conflicts from recent commits.
>
> I have reviewed v22-0022* patch and I have some comments.
>
> 1.
> >It also changes to the column names
On Wed, Jan 25, 2023 at 8:42 AM Masahiko Sawada
wrote:
>
> On Mon, Jan 23, 2023 at 8:20 PM John Naylor
> wrote:
> >
> > On Mon, Jan 16, 2023 at 3:18 PM Masahiko Sawada
wrote:
> > >
> > > On Mon, Jan 16, 2023 at 2:02 PM John Naylor
> > > wrote:
> >
> > In v21, all of your v20 improvements to
On Mon, Jan 23, 2023 at 8:20 PM John Naylor
wrote:
>
> On Mon, Jan 16, 2023 at 3:18 PM Masahiko Sawada wrote:
> >
> > On Mon, Jan 16, 2023 at 2:02 PM John Naylor
> > wrote:
>
> In v21, all of your v20 improvements to the radix tree template and test have
> been squashed into 0003, with one
On Mon, Jan 23, 2023 at 6:00 PM John Naylor
wrote:
>
> Attached is a rebase to fix conflicts from recent commits.
I have reviewed v22-0022* patch and I have some comments.
1.
>It also changes to the column names max_dead_tuples and num_dead_tuples and to
>show the progress information in bytes.
On Tue, Jan 17, 2023 at 8:06 PM John Naylor
wrote:
>
> On Mon, Jan 16, 2023 at 3:18 PM Masahiko Sawada wrote:
> >
> > On Mon, Jan 16, 2023 at 2:02 PM John Naylor
> > wrote:
>
> Attached is an update that mostly has the modest goal of getting CI green
> again. v19-0003 has squashed the entire
On Mon, Jan 16, 2023 at 3:18 PM Masahiko Sawada
wrote:
>
> On Mon, Jan 16, 2023 at 2:02 PM John Naylor
> wrote:
> > + * Add Tids on a block to TidStore. The caller must ensure the offset
numbers
> > + * in 'offsets' are ordered in ascending order.
> >
> > Must? What happens otherwise?
>
> It
On Mon, Jan 16, 2023 at 3:18 PM Masahiko Sawada
wrote:
>
> On Mon, Jan 16, 2023 at 2:02 PM John Naylor
> wrote:
> >
> Thanks! cfbot complaints about some warnings but these are expected
> (due to unused delete routines etc). But one reported error[1] might
> be relevant with 0002 patch?
>
On Mon, Jan 16, 2023 at 2:02 PM John Naylor
wrote:
>
> On Thu, Jan 12, 2023 at 9:51 PM Masahiko Sawada wrote:
> >
> > On Thu, Jan 12, 2023 at 5:21 PM John Naylor
> > wrote:
> > >
> > > Okay, I'll squash the previous patch and work on cleaning up the
> > > internals. I'll keep the external APIs
On Fri, Dec 23, 2022 at 4:33 PM John Naylor
wrote:
>
>
>
> On Thu, Dec 22, 2022 at 10:00 PM Masahiko Sawada
> wrote:
>
> > If the value is a power of 2, it seems to work perfectly fine. But for
> > example if it's 700MB, the total memory exceeds the limit:
> >
> > 2*(1+2+4+8+16+32+64+128) =
On Thu, Jan 12, 2023 at 5:21 PM John Naylor
wrote:
>
> On Thu, Jan 12, 2023 at 12:44 PM Masahiko Sawada
> wrote:
> >
> > On Wed, Jan 11, 2023 at 12:13 PM John Naylor
> > wrote:
> > >
> > > On Tue, Jan 10, 2023 at 7:08 PM Masahiko Sawada
> > > wrote:
>
> > I agree to keep this as a template.
On Thu, Jan 12, 2023 at 12:44 PM Masahiko Sawada
wrote:
>
> On Wed, Jan 11, 2023 at 12:13 PM John Naylor
> wrote:
> >
> > On Tue, Jan 10, 2023 at 7:08 PM Masahiko Sawada
wrote:
> I agree to keep this as a template.
Okay, I'll squash the previous patch and work on cleaning up the internals.
On Wed, Jan 11, 2023 at 12:13 PM John Naylor
wrote:
>
> On Tue, Jan 10, 2023 at 7:08 PM Masahiko Sawada wrote:
>
> > It looks no problem in terms of vacuum integration, although I've not
> > fully tested yet. TID store uses the radix tree as the main storage,
> > and with the template radix
I wrote:
> I see. IIUC from a brief re-reading of the code, saving that chunk would
only save us from re-loading "parent->shift" from L1 cache and shifting the
key. The cycles spent doing that seem small compared to the rest of the
work involved in growing a node. Expressions like "if (idx < 0)
On Tue, Jan 10, 2023 at 7:08 PM Masahiko Sawada
wrote:
> It looks no problem in terms of vacuum integration, although I've not
> fully tested yet. TID store uses the radix tree as the main storage,
> and with the template radix tree, the data types for shared and
> non-shared will be different.
On Mon, Jan 9, 2023 at 5:59 PM John Naylor wrote:
>
> > [working on templating]
>
> In the end, I decided to base my effort on v8, and not v12 (based on one of
> my less-well-thought-out ideas). The latter was a good experiment, but it did
> not lead to an increase in readability as I had
On Tue, Dec 27, 2022 at 2:24 PM John Naylor
wrote:
>
> On Tue, Dec 27, 2022 at 12:14 AM Masahiko Sawada
> wrote:
> >
> > On Fri, Dec 23, 2022 at 8:47 PM John Naylor
> > wrote:
>
> > These 4 patches make sense to me.We can merge them into 0002 patch
>
> Okay, then I'll squash them when I post
On Tue, Dec 27, 2022 at 12:14 AM Masahiko Sawada
wrote:
>
> On Fri, Dec 23, 2022 at 8:47 PM John Naylor
> wrote:
> These 4 patches make sense to me.We can merge them into 0002 patch
Okay, then I'll squash them when I post my next patch.
> and I'll do similar changes for functions for leaf
On Fri, Dec 23, 2022 at 8:47 PM John Naylor
wrote:
>
> I wrote:
>
> > - Try templating out the differences between local and shared memory.
>
> Here is a brief progress report before Christmas vacation.
Thanks!
>
> I thought the best way to approach this was to go "inside out", that is,
>
On Thu, Dec 22, 2022 at 10:00 PM Masahiko Sawada
wrote:
> If the value is a power of 2, it seems to work perfectly fine. But for
> example if it's 700MB, the total memory exceeds the limit:
>
> 2*(1+2+4+8+16+32+64+128) = 510MB (72.8% of 700MB) -> keep going
> 510 + 256 = 766MB -> stop but it
On Thu, Dec 22, 2022 at 7:24 PM John Naylor
wrote:
>
>
> On Wed, Dec 21, 2022 at 3:09 PM Masahiko Sawada wrote:
> >
> > On Tue, Dec 20, 2022 at 3:09 PM John Naylor
> > wrote:
>
> > > https://www.postgresql.org/message-id/20220704211822.kfxtzpcdmslzm2dy%40awork3.anarazel.de
> > >
> > > I'm
On Wed, Dec 21, 2022 at 3:09 PM Masahiko Sawada
wrote:
>
> On Tue, Dec 20, 2022 at 3:09 PM John Naylor
> wrote:
> >
https://www.postgresql.org/message-id/20220704211822.kfxtzpcdmslzm2dy%40awork3.anarazel.de
> >
> > I'm guessing the hash join case can afford to be precise about memory
because it
On Tue, Dec 20, 2022 at 3:09 PM John Naylor
wrote:
>
>
> On Mon, Dec 19, 2022 at 2:14 PM Masahiko Sawada wrote:
> >
> > On Tue, Dec 13, 2022 at 1:04 AM Masahiko Sawada
> > wrote:
>
> > > Looking at other code using DSA such as tidbitmap.c and nodeHash.c, it
> > > seems that they look at only
On Mon, Dec 19, 2022 at 2:14 PM Masahiko Sawada
wrote:
>
> On Tue, Dec 13, 2022 at 1:04 AM Masahiko Sawada
wrote:
> > Looking at other code using DSA such as tidbitmap.c and nodeHash.c, it
> > seems that they look at only memory that are actually dsa_allocate'd.
> > To be exact, we estimate the
On Mon, Dec 12, 2022 at 7:14 PM John Naylor
wrote:
>
>
> On Fri, Dec 9, 2022 at 8:33 PM Masahiko Sawada wrote:
> >
> > On Fri, Dec 9, 2022 at 5:53 PM John Naylor
> > wrote:
> > >
>
> > > I don't think that'd be very controversial, but I'm also not sure why
> > > we'd need 4MB -- can you
On Fri, Dec 9, 2022 at 8:33 PM Masahiko Sawada
wrote:
>
> On Fri, Dec 9, 2022 at 5:53 PM John Naylor
wrote:
> >
> > I don't think that'd be very controversial, but I'm also not sure why
we'd need 4MB -- can you explain in more detail what exactly we'd need so
that the feature would work? (The
On Fri, Dec 9, 2022 at 5:53 PM John Naylor wrote:
>
>
> On Fri, Dec 9, 2022 at 8:20 AM Masahiko Sawada wrote:
>
> > In the meanwhile, I've been working on vacuum integration. There are
> > two things I'd like to discuss some time:
> >
> > The first is the minimum of maintenance_work_mem, 1 MB.
On Fri, Dec 9, 2022 at 8:20 AM Masahiko Sawada
wrote:
> In the meanwhile, I've been working on vacuum integration. There are
> two things I'd like to discuss some time:
>
> The first is the minimum of maintenance_work_mem, 1 MB. Since the
> initial DSA segment size is 1MB
On Tue, Dec 6, 2022 at 7:32 PM John Naylor wrote:
>
> On Fri, Dec 2, 2022 at 11:42 PM Masahiko Sawada wrote:
> >
> > > On Mon, Nov 14, 2022 at 7:59 PM John Naylor
> > > wrote:
> > > >
> > > > - Optimize node128 insert.
> > >
> > > I've attached a rough start at this. The basic idea is borrowed
On Thu, Dec 1, 2022 at 3:03 PM Masahiko Sawada
wrote:
>
> On Thu, Dec 1, 2022 at 4:00 PM John Naylor
wrote:
> >
> > The bigger question in my mind is: Why is there an atomic variable in
backend-local memory?
>
> Because I use the same radix_tree and radix_tree_control structs for
> non-parallel
On Thu, Dec 1, 2022 at 4:00 PM John Naylor wrote:
>
>
> On Wed, Nov 30, 2022 at 11:09 PM Masahiko Sawada
> wrote:
> >
> > I've investigated this issue and have a question about using atomic
> > variables on palloc'ed memory. In non-parallel vacuum cases,
> > radix_tree_control is allocated via
On Wed, Nov 30, 2022 at 2:28 PM Masahiko Sawada
wrote:
>
> On Fri, Nov 25, 2022 at 5:00 PM John Naylor
> wrote:
> > These hacks were my work, but I think we can improve that by having two
versions of NODE_HAS_FREE_SLOT -- one for fixed- and one for variable-sized
nodes. For that to work, in
On Wed, Nov 30, 2022 at 11:09 PM Masahiko Sawada
wrote:
>
> I've investigated this issue and have a question about using atomic
> variables on palloc'ed memory. In non-parallel vacuum cases,
> radix_tree_control is allocated via aset.c. IIUC in 32-bit machines,
> the memory allocated by aset.c is
On Wed, Nov 23, 2022 at 2:10 AM Andres Freund wrote:
>
> On 2022-11-21 17:06:56 +0900, Masahiko Sawada wrote:
> > Sure. I've attached the v10 patches. 0004 is the pure refactoring
> > patch and 0005 patch introduces the pointer tagging.
>
> This failed on cfbot, with som many crashes that the VM
On Tue, Nov 29, 2022 at 1:36 PM John Naylor
wrote:
>
> While creating a benchmark for inserting into node128-inner, I found a bug.
> If a caller deletes from a node128, the slot index is set to invalid, but the
> child pointer is still valid. Do that a few times, and every child pointer is
>
On Fri, Nov 25, 2022 at 6:47 PM John Naylor
wrote:
>
>
>
> On Thu, Nov 24, 2022 at 9:54 PM Masahiko Sawada wrote:
> >
> > [v11]
>
> There is one more thing that just now occurred to me: In expanding the use of
> size classes, that makes rebasing and reworking the shared memory piece more
>
On Fri, Nov 25, 2022 at 5:00 PM John Naylor
wrote:
>
> On Thu, Nov 24, 2022 at 9:54 PM Masahiko Sawada wrote:
> >
> > So it seems that there are two candidates of rt_node structure: (1)
> > all nodes except for node256 are variable-size nodes and use pointer
> > tagging, and (2) node32 and
There are a few things up in the air, so I'm coming back to this list to
summarize and add a recent update:
On Mon, Nov 14, 2022 at 7:59 PM John Naylor
wrote:
>
> - See how much performance we actually gain from tagging the node kind.
Needs a benchmark that has enough branch mispredicts and
201 - 300 of 435 matches
Mail list logo
