proposal: Allocate work_mem From Pool

2022-07-10 Thread Joseph D Wagner
I'm new here, so forgive me if this is a bad idea or my lack of knowledge on how to optimize PostgreSQL. I find PostgreSQL to be great with a large number of small transactions, which covers most use cases. However, my experience has not been so great on the opposite end -- a small number of larg

Re: proposal: Allocate work_mem From Pool

2022-07-10 Thread Justin Pryzby
On Sun, Jul 10, 2022 at 08:45:38PM -0700, Joseph D Wagner wrote: > However, that's risky because it's 3GB per operation, not per > query/connection; it could easily spiral out of control. This is a well-known deficiency. I suggest to dig up the old threads to look into. It's also useful to includ

RE: proposal: Allocate work_mem From Pool

2022-07-12 Thread Joseph D Wagner
>> I think it would be better if work_mem was allocated from a pool >> of memory > I think this has been proposed before, and the issue/objection > with this idea is probably that query plans will be inconsistent, > and end up being sub-optimal. > work_mem is considered at planning time, but I t

Re: proposal: Allocate work_mem From Pool

2022-07-12 Thread John Naylor
On Tue, Jul 12, 2022 at 5:55 PM Joseph D Wagner wrote: > Before I try to answer that, I need to know how the scheduler works. As I understand the term used, there is no scheduler inside Postgres for user connections -- they're handled by the OS kernel. That's probably why it'd be a difficult proj

Re: proposal: Allocate work_mem From Pool

2022-07-12 Thread Justin Pryzby
On Tue, Jul 12, 2022 at 03:55:39AM -0700, Joseph D Wagner wrote: > Before I try to answer that, I need to know how the scheduler works. > > Let's say there's a max of 8 worker process, and 12 queries trying to run. > When does query #9 run? After the first of 1-8 completes, simple FIFO? > Or somet

Re: proposal: Allocate work_mem From Pool

2022-07-12 Thread Joseph D Wagner
Before I try to answer that, I need to know how the scheduler works. As I understand the term used, there is no scheduler inside Postgres for user connections -- they're handled by the OS kernel. Then, I'm probably using the wrong term. Right now, I have max_worker_processes set to 16. What h

Re: proposal: Allocate work_mem From Pool

2022-07-13 Thread Justin Pryzby
On Tue, Jul 12, 2022 at 08:49:10PM -0700, Joseph D Wagner wrote: > > > Before I try to answer that, I need to know how the scheduler works. > > > As I understand the term used, there is no scheduler inside Postgres > > for user connections -- they're handled by the OS kernel. > > Then, I'm probab