Re: scalability bottlenecks with (many) partitions (and more)

2024-06-25 Thread Robert Haas
On Tue, Jun 25, 2024 at 6:04 AM Tomas Vondra wrote: > Yeah, definitely needs comment explaining this. > > I admit those numbers are pretty arbitrary primes, to implement a > trivial hash function. That was good enough for a PoC patch, but maybe > for a "proper" version this should use a better

Re: scalability bottlenecks with (many) partitions (and more)

2024-06-25 Thread Tomas Vondra
On 6/24/24 17:05, Robert Haas wrote: > On Sun, Jan 28, 2024 at 4:57 PM Tomas Vondra > wrote: >> For NUM_LOCK_PARTITIONS this is pretty simple (see 0001 patch). The >> LWLock table has 16 partitions by default - it's quite possible that on >> machine with many cores and/or many partitions, we

Re: scalability bottlenecks with (many) partitions (and more)

2024-06-24 Thread Robert Haas
On Sun, Jan 28, 2024 at 4:57 PM Tomas Vondra wrote: > For NUM_LOCK_PARTITIONS this is pretty simple (see 0001 patch). The > LWLock table has 16 partitions by default - it's quite possible that on > machine with many cores and/or many partitions, we can easily hit this. > So I bumped this 4x to 64

Re: scalability bottlenecks with (many) partitions (and more)

2024-01-31 Thread Tomas Vondra
On 1/29/24 16:42, Ronan Dunklau wrote: > Le lundi 29 janvier 2024, 15:59:04 CET Tomas Vondra a écrit : >> I'm not sure work_mem is a good parameter to drive this. It doesn't say >> how much memory we expect the backend to use - it's a per-operation >> limit, so it doesn't work particularly well

Re: scalability bottlenecks with (many) partitions (and more)

2024-01-29 Thread Ronan Dunklau
Le lundi 29 janvier 2024, 15:59:04 CET Tomas Vondra a écrit : > I'm not sure work_mem is a good parameter to drive this. It doesn't say > how much memory we expect the backend to use - it's a per-operation > limit, so it doesn't work particularly well with partitioning (e.g. with > 100 partitions,

Re: scalability bottlenecks with (many) partitions (and more)

2024-01-29 Thread Tomas Vondra
On 1/29/24 15:15, Ronan Dunklau wrote: > Le lundi 29 janvier 2024, 13:17:07 CET Tomas Vondra a écrit : >>> Did you try running an strace on the process ? That may give you some >>> hindsights into what malloc is doing. A more sophisticated approach would >>> be using stap and plugging it into

Re: scalability bottlenecks with (many) partitions (and more)

2024-01-29 Thread Ronan Dunklau
Le lundi 29 janvier 2024, 13:17:07 CET Tomas Vondra a écrit : > > Did you try running an strace on the process ? That may give you some > > hindsights into what malloc is doing. A more sophisticated approach would > > be using stap and plugging it into the malloc probes, for example > >

Re: scalability bottlenecks with (many) partitions (and more)

2024-01-29 Thread Tomas Vondra
On 1/29/24 09:53, Ronan Dunklau wrote: > Le dimanche 28 janvier 2024, 22:57:02 CET Tomas Vondra a écrit : > > Hi Tomas ! > > I'll comment on glibc-malloc part as I studied that part last year, and > proposed some things here: https://www.postgresql.org/message-id/ >

Re: scalability bottlenecks with (many) partitions (and more)

2024-01-29 Thread Ronan Dunklau
Le dimanche 28 janvier 2024, 22:57:02 CET Tomas Vondra a écrit : Hi Tomas ! I'll comment on glibc-malloc part as I studied that part last year, and proposed some things here: https://www.postgresql.org/message-id/ 3424675.QJadu78ljV%40aivenlaptop > FWIW where does the malloc overhead come