Hello Anton,
14.03.2024 23:36, Anton A. Melnikov wrote:
On 13.03.2024 10:41, Anton A. Melnikov wrote:
Here is a version updated for the current master.
During patch updating i mistakenly added double counting of deallocatated
blocks.
That's why the tests in the patch tester failed.
Fixed
On 13.03.2024 10:41, Anton A. Melnikov wrote:
Here is a version updated for the current master.
During patch updating i mistakenly added double counting of deallocatated
blocks.
That's why the tests in the patch tester failed.
Fixed it and squashed fix 0002 with 0001.
Here is fixed version.
On 12.03.2024 16:30, Aleksander Alekseev wrote:
Just wanted to let you know that v20231226 doesn't apply. The patch
needs love from somebody interested in it.
Thanks for pointing to this!
Here is a version updated for the current master.
With the best regards,
--
Anton A. Melnikov
Postgres
Hi,
> I took a closer look at this patch over the last couple days, and I did
> a bunch of benchmarks to see how expensive the accounting is. The good
> news is that I haven't observed any overhead - I did two simple tests,
> that I think would be particularly sensitive to this:
>
> [...]
Just
Hi,
I wanted to take a look at the patch, and I noticed it's broken since
3d51cb5197 renamed a couple pgstat functions in August. I plan to maybe
do some benchmarks etc. preferably on current master, so here's a
version fixing that minor bitrot.
As for the patch, I only skimmed through the
On 12/26/23 11:49, Anton A. Melnikov wrote:
> Hello!
>
> Earlier in this thread, the pgbench results were published, where with a
> strong memory limit of 100MB
> a significant, about 10%, decrease in TPS was observed [1].
>
> Using dedicated server with 12GB RAM and methodology described in
hi.
+static void checkAllocations();
should be "static void checkAllocations(void);" ?
PgStatShared_Memtrack there is a lock, but seems not initialized, and
not used. Can you expand on it?
So in view pg_stat_global_memory_tracking, column
"total_memory_reserved" is a point of time, total memory
Hi,
On 2023-11-07 15:55:48 -0500, Stephen Frost wrote:
> * Andres Freund (and...@anarazel.de) wrote:
> > On 2023-11-06 13:02:50 -0500, Stephen Frost wrote:
> > > > > The max_total_memory limit is checked whenever the global counters are
> > > > > updated. There is no special error handling if a
Greetings,
* Andres Freund (and...@anarazel.de) wrote:
> On 2023-11-06 13:02:50 -0500, Stephen Frost wrote:
> > > > The max_total_memory limit is checked whenever the global counters are
> > > > updated. There is no special error handling if a memory allocation
> > > > exceeds
> > > > the global
Hi,
On 2023-11-06 13:02:50 -0500, Stephen Frost wrote:
> > > The max_total_memory limit is checked whenever the global counters are
> > > updated. There is no special error handling if a memory allocation exceeds
> > > the global limit. That allocation returns a NULL for malloc style
> > >
Greetings,
* Andres Freund (and...@anarazel.de) wrote:
> On 2023-10-31 17:11:26 +, John Morris wrote:
> > Postgres memory reservations come from multiple sources.
> >
> > * Malloc calls made by the Postgres memory allocators.
> > * Static shared memory created by the postmaster at
Hi,
On 2023-10-31 17:11:26 +, John Morris wrote:
> Postgres memory reservations come from multiple sources.
>
> * Malloc calls made by the Postgres memory allocators.
> * Static shared memory created by the postmaster at server startup,
> * Dynamic shared memory created by the
Hi,
On 2023-10-24 09:39:42 +0700, Andrei Lepikhov wrote:
> On 20/10/2023 19:39, Stephen Frost wrote:
> Greetings,
> > * Andrei Lepikhov (a.lepik...@postgrespro.ru) wrote:
> > > The only issue I worry about is the uncertainty and clutter that can be
> > > created by this feature. In the worst
On 20/10/2023 19:39, Stephen Frost wrote:
Greetings,
* Andrei Lepikhov (a.lepik...@postgrespro.ru) wrote:
The only issue I worry about is the uncertainty and clutter that can be
created by this feature. In the worst case, when we have a complex error
stack (including the extension's CATCH
Greetings,
* Andrei Lepikhov (a.lepik...@postgrespro.ru) wrote:
> The only issue I worry about is the uncertainty and clutter that can be
> created by this feature. In the worst case, when we have a complex error
> stack (including the extension's CATCH sections, exceptions in stored
>
On 20/10/2023 05:06, Stephen Frost wrote:
Greetings,
* Andrei Lepikhov (a.lepik...@postgrespro.ru) wrote:
On 19/10/2023 02:00, Stephen Frost wrote:
* Andrei Lepikhov (a.lepik...@postgrespro.ru) wrote:
On 29/9/2023 09:52, Andrei Lepikhov wrote:
On 22/5/2023 22:59,
Greetings,
* Andres Freund (and...@anarazel.de) wrote:
> On 2023-10-19 18:06:10 -0400, Stephen Frost wrote:
> > Ignoring such would defeat much of the point of this effort- which is to
> > get to a position where we can say with some confidence that we're not
> > going to go over some limit that
Hi,
On 2023-10-19 18:06:10 -0400, Stephen Frost wrote:
> Ignoring such would defeat much of the point of this effort- which is to
> get to a position where we can say with some confidence that we're not
> going to go over some limit that the user has set and therefore not
> allow ourselves to end
Greetings,
* Andrei Lepikhov (a.lepik...@postgrespro.ru) wrote:
> On 19/10/2023 02:00, Stephen Frost wrote:
> > * Andrei Lepikhov (a.lepik...@postgrespro.ru) wrote:
> > > On 29/9/2023 09:52, Andrei Lepikhov wrote:
> > > > On 22/5/2023 22:59, reid.thomp...@crunchydata.com wrote:
> > > > > Attach
On 19/10/2023 02:00, Stephen Frost wrote:
Greetings,
* Andrei Lepikhov (a.lepik...@postgrespro.ru) wrote:
On 29/9/2023 09:52, Andrei Lepikhov wrote:
On 22/5/2023 22:59, reid.thomp...@crunchydata.com wrote:
Attach patches updated to master.
Pulled from patch 2 back to patch 1 a change that
Greetings,
* Andrei Lepikhov (a.lepik...@postgrespro.ru) wrote:
> On 29/9/2023 09:52, Andrei Lepikhov wrote:
> > On 22/5/2023 22:59, reid.thomp...@crunchydata.com wrote:
> > > Attach patches updated to master.
> > > Pulled from patch 2 back to patch 1 a change that was also pertinent
> > > to
On 29/9/2023 09:52, Andrei Lepikhov wrote:
On 22/5/2023 22:59, reid.thomp...@crunchydata.com wrote:
Attach patches updated to master.
Pulled from patch 2 back to patch 1 a change that was also pertinent
to patch 1.
+1 to the idea, have doubts on the implementation.
I have a question. I see
On 22/5/2023 22:59, reid.thomp...@crunchydata.com wrote:
Attach patches updated to master.
Pulled from patch 2 back to patch 1 a change that was also pertinent to patch 1.
+1 to the idea, have doubts on the implementation.
I have a question. I see the feature triggers ERROR on the exceeding of
On Mon, 2023-05-22 at 08:42 -0400, reid.thomp...@crunchydata.com wrote:
More followup to the above.
>
> I experimented on my system regarding
> "The simple query select * from generate_series(0, 1000) shows roughly
> 18.9 % degradation on my test server."
>
> My laptop:
> 32GB ram
> 11th
On Mon, 2023-05-22 at 08:42 -0400, reid.thomp...@crunchydata.com wrote:
> On Wed, 2023-05-17 at 23:07 -0400, reid.thomp...@crunchydata.com wrote:
> > Thanks for the feedback.
> >
> > I'm plannning to look at this.
> >
> > Is your benchmark something that I could utilize? I.E. is it a set of
> >
On Wed, 2023-05-17 at 23:07 -0400, reid.thomp...@crunchydata.com wrote:
> Thanks for the feedback.
>
> I'm plannning to look at this.
>
> Is your benchmark something that I could utilize? I.E. is it a set of
> scripts or a standard test from somewhere that I can duplicate?
>
> Thanks,
> Reid
>
On Wed, 2023-04-19 at 23:28 +, Arne Roland wrote:
> > Thank you! I just tried our benchmark and got a performance
> > degration > of around 28 %, which is way better than the last
> > patch.
> >
> > The simple query select * from generate_series(0, 1000) shows >
> > roughly 18.9 %
Thank you! I just tried our benchmark and got a performance degration of around
28 %, which is way better than the last patch.
The simple query select * from generate_series(0, 1000) shows roughly 18.9
% degradation on my test server.
By raising initial_allocation_allowance and
Updated patches attached.
Rebased to current master.
Added additional columns to pg_stat_global_memory_allocation to summarize
backend allocations by type.
Updated documentation.
Corrected some issues noted in review by John Morris.
Added code re EXEC_BACKEND for dev-max-memory branch.
From
On Thu, 2023-03-30 at 16:11 +, John Morris wrote:
> Hi Reid!
> Some thoughts
> I was looking at lmgr/proc.c, and I see a potential integer overflow
> - bothmax_total_bkend_mem and result are declared as “int”, so the
> expression “max_total_bkend_mem * 1024 * 1024 - result * 1024 * 1024”
>
Hi Reid!
Some thoughts
I was looking at lmgr/proc.c, and I see a potential integer overflow - both
max_total_bkend_mem and result are declared as “int”, so the expression
“max_total_bkend_mem * 1024 * 1024 - result * 1024 * 1024” could have a problem
when max_total_bkend_mem is set to 2G or
Updated patches attached.
pg-stat-activity-backend-memory-allocated
DSM allocations created by a process and not destroyed prior to it's exit are
considered
On 2023-03-02 14:41:26 -0500, reid.thomp...@crunchydata.com wrote:
> Patch has been rebased to master.
Quite a few prior review comments seem to not have been
addressed. There's not much point in posting new versions without that.
I think there's zero chance 0002 can make it into 16. If 0001 is
On Mon, 2023-02-13 at 16:26 -0800, Andres Freund wrote:
> Hi,
>
> The tests recently started to fail:
>
> https://cirrus-ci.com/github/postgresql-cfbot/postgresql/commitfest%2F42%2F3867
>
> I marked this as waiting on author.
>
> Greetings,
>
> Andres Freund
Patch has been rebased to master.
Hi,
On 2023-01-26 15:27:20 -0500, Reid Thompson wrote:
> Yes, just a rebase. There is still work to be done per earlier in the
> thread.
The tests recently started to fail:
https://cirrus-ci.com/github/postgresql-cfbot/postgresql/commitfest%2F42%2F3867
> I do want to follow up and note re
Regarding the shared counter noted here,
> What you could do is to have a single, imprecise, shared counter for the total
> memory allocation, and have a backend-local "allowance". When the allowance is
> used up, refill it from the shared counter (a single atomic op).
Is there a preferred or
On Mon, 2023-01-23 at 12:31 -0800, Andres Freund wrote:
> Hi,
>
> I think it's basically still waiting on author, until the O(N) cost is gone
> from the overflow limit check.
>
> Greetings,
>
> Andres Freund
Yes, just a rebase. There is still work to be done per earlier in the
thread.
I do
Hi,
On 2023-01-23 10:48:38 -0500, Reid Thompson wrote:
> On Thu, 2023-01-19 at 16:50 +0530, vignesh C wrote:
> >
> > The patch does not apply on top of HEAD as in [1], please post a rebased
> > patch:
> >
> > Regards,
> > Vignesh
>
> rebased patch attached
I think it's basically still
On Thu, 2023-01-19 at 16:50 +0530, vignesh C wrote:
>
> The patch does not apply on top of HEAD as in [1], please post a rebased
> patch:
>
> Regards,
> Vignesh
rebased patch attached
Thanks,
Reid
From b32a346d6e0e00c568e9a285ad15fc2703998c26 Mon Sep 17 00:00:00 2001
From: Reid Thompson
On Fri, 6 Jan 2023 at 00:19, Reid Thompson
wrote:
>
> On Tue, 2023-01-03 at 16:22 +0530, vignesh C wrote:
> >
> > The patch does not apply on top of HEAD as in [1], please post a
> > rebased patch:
> > ...
> > Regards,
> > Vignesh
> >
>
> Attached is rebased patch, with some updates related
Hi,
On 2023-01-13 09:15:10 -0500, Reid Thompson wrote:
> On Mon, 2023-01-09 at 18:31 -0800, Andres Freund wrote:
> > > Dynamic shared memory allocations are included only in the value displayed
> > > for the backend that created them, they are not included in the value for
> > > backends that are
On Mon, 2023-01-09 at 18:31 -0800, Andres Freund wrote:
> Hi,
>
> On 2023-01-05 13:44:20 -0500, Reid Thompson wrote:
> > This new field displays the current bytes of memory allocated to the
> > backend process. It is updated as memory for the process is
> > malloc'd/free'd. Memory allocated to
Hi,
On 2023-01-05 13:44:20 -0500, Reid Thompson wrote:
> From 0a6b152e0559a25033bd7d43eb0959432e0d Mon Sep 17 00:00:00 2001
> From: Reid Thompson
> Date: Thu, 11 Aug 2022 12:01:25 -0400
> Subject: [PATCH 1/2] Add tracking of backend memory allocated to
> pg_stat_activity
>
> This new field
On Tue, 2023-01-03 at 16:22 +0530, vignesh C wrote:
>
> The patch does not apply on top of HEAD as in [1], please post a
> rebased patch:
> ...
> Regards,
> Vignesh
>
Attached is rebased patch, with some updates related to committed changes.
Thanks,
Reid
--
Reid Thompson
Senior Software
On Fri, 9 Dec 2022 at 20:41, Reid Thompson
wrote:
>
> On Tue, 2022-12-06 at 10:32 -0800, Andres Freund wrote:
> > Hi,
> >
> > On 2022-11-26 22:22:15 -0500, Reid Thompson wrote:
> > > rebased/patched to current master && current pg-stat-activity-
> > > backend-memory-allocated
> >
> > This version
On Tue, 2022-12-06 at 10:32 -0800, Andres Freund wrote:
> Hi,
>
> On 2022-11-26 22:22:15 -0500, Reid Thompson wrote:
> > rebased/patched to current master && current pg-stat-activity-
> > backend-memory-allocated
>
> This version fails to build with msvc, and builds with warnings on
> other
>
Hi,
On 2022-11-26 22:22:15 -0500, Reid Thompson wrote:
> rebased/patched to current master && current
> pg-stat-activity-backend-memory-allocated
This version fails to build with msvc, and builds with warnings on other
platforms.
https://cirrus-ci.com/build/5410696721072128
msvc:
[20:26:51.286]
On Thu, 2022-11-03 at 11:48 -0400, Reid Thompson wrote:
> On Tue, 2022-10-25 at 11:49 -0400, Reid Thompson wrote:
>
> Rebased to current. Add a couple changes per conversation with D
> Christensen (include units in field name, group field with
> backend_xid
> and backend_xmin fields in
On Tue, 2022-10-25 at 11:49 -0400, Reid Thompson wrote:
> Hi Arne,
>
> On Mon, 2022-10-24 at 15:27 +, Arne Roland wrote:
> > Hello Reid,
> >
> > could you rebase the patch again? It doesn't apply currently
> > (http://cfbot.cputube.org/patch_40_3867.log). Thanks!
>
> rebased patches
Hi Arne,
On Mon, 2022-10-24 at 15:27 +, Arne Roland wrote:
> Hello Reid,
>
> could you rebase the patch again? It doesn't apply currently
> (http://cfbot.cputube.org/patch_40_3867.log). Thanks!
rebased patches attached.
> You mention, that you want to prevent the compiler from getting
>
Hello Reid,
could you rebase the patch again? It doesn't apply currently
(http://cfbot.cputube.org/patch_40_3867.log). Thanks!
You mention, that you want to prevent the compiler from getting cute.
I don't think this comments are exactly helpful in the current state. I think
probably fine to
On Thu, 2022-09-15 at 12:07 +0400, Ibrar Ahmed wrote:
>
> The patch does not apply; please rebase the patch.
>
> patching file src/backend/utils/misc/guc.c
> Hunk #1 FAILED at 3664.
> 1 out of 1 hunk FAILED -- saving rejects to file
> src/backend/utils/misc/guc.c.rej
>
> patching file
On Mon, Sep 12, 2022 at 8:30 PM Reid Thompson
wrote:
> On Fri, 2022-09-09 at 12:14 -0500, Justin Pryzby wrote:
> > On Sat, Sep 03, 2022 at 11:40:03PM -0400, Reid Thompson wrote:
> > > > > + 0, 0, INT_MAX,
> > > > > + NULL, NULL, NULL
> > > > I think this needs a
On Fri, 2022-09-09 at 12:14 -0500, Justin Pryzby wrote:
> On Sat, Sep 03, 2022 at 11:40:03PM -0400, Reid Thompson wrote:
> > > > + 0, 0, INT_MAX,
> > > > + NULL, NULL, NULL
> > > I think this needs a maximum like INT_MAX/1024/1024
> >
> > Is this noting that we'd set a
On Sat, Sep 03, 2022 at 11:40:03PM -0400, Reid Thompson wrote:
> > > + 0, 0, INT_MAX,
> > > + NULL, NULL, NULL
> > I think this needs a maximum like INT_MAX/1024/1024
>
> Is this noting that we'd set a ceiling of 2048MB?
The reason is that you're later multiplying it
Greetings,
* David Rowley (dgrowle...@gmail.com) wrote:
> On Thu, 1 Sept 2022 at 04:52, Reid Thompson
> wrote:
> > Add the ability to limit the amount of memory that can be allocated to
> > backends.
>
> Are you aware that relcache entries are stored in backend local memory
> and that once
On Fri, 2022-09-02 at 09:30 +0200, Drouvot, Bertrand wrote:
> Hi,
>
> I'm not sure we are choosing the right victims here (aka the ones
> that are doing the request that will push the total over the limit).
>
> Imagine an extreme case where a single backend consumes say 99% of
> the limit,
On Thu, 2022-09-01 at 11:48 +0900, Kyotaro Horiguchi wrote:
> >
> > The patch seems to limit both of memory-context allocations and DSM
> > allocations happen on a specific process by the same budget. In the
> > fist place I don't think it's sensible to cap the amount of DSM
> > allocations by
On Wed, 2022-08-31 at 12:34 -0500, Justin Pryzby wrote:
> You should name the patches with different prefixes, like
> 001,002,003 Otherwise, cfbot may try to apply them in the wrong
> order.
> git format-patch is the usual tool for that.
Thanks for the pointer. My experience with git in the past
On Thu, 1 Sept 2022 at 04:52, Reid Thompson
wrote:
> Add the ability to limit the amount of memory that can be allocated to
> backends.
Are you aware that relcache entries are stored in backend local memory
and that once we've added a relcache entry for a relation that we have
no current code
At Wed, 31 Aug 2022 12:50:19 -0400, Reid Thompson
wrote in
> Hi Hackers,
>
> Add the ability to limit the amount of memory that can be allocated to
> backends.
The patch seems to limit both of memory-context allocations and DSM
allocations happen on a specific process by the same budget. In
On Wed, Aug 31, 2022 at 12:50:19PM -0400, Reid Thompson wrote:
> Hi Hackers,
>
> Add the ability to limit the amount of memory that can be allocated to
> backends.
>
> This builds on the work that adds backend memory allocated to
> pg_stat_activity
>
Hi Hackers,
Add the ability to limit the amount of memory that can be allocated to
backends.
This builds on the work that adds backend memory allocated to
pg_stat_activity
https://www.postgresql.org/message-id/67bb5c15c0489cb499723b0340f16e10c22485ec.camel%40crunchydata.com
Both patches are
63 matches
Mail list logo
