Re: Why is Postgres only using 8 cores for partitioned count? [Parallel Append]

2021-02-15 Thread David Rowley
On Tue, 16 Feb 2021 at 02:12, Fabio Pardi wrote: > > On 14/02/2021 22:16, Gavin Flower wrote: > > While I agree it might be good to be able specify the number of workers, > > sure it would be possible to derive a suitable default based on the number > > of effective processors available? > > I

Re: Why is Postgres only using 8 cores for partitioned count? [Parallel Append]

2021-02-15 Thread David Rowley
On Mon, 15 Feb 2021 at 10:16, Gavin Flower wrote: > Just wondering why there is a hard coded limit. I don't see where the hardcoded limit is. The total number is limited to max_parallel_workers_per_gather, but there's nothing hardcoded about the value of that. > While I agree it might be good

Re: Why is Postgres only using 8 cores for partitioned count? [Parallel Append]

2021-02-15 Thread Fabio Pardi
On 14/02/2021 22:16, Gavin Flower wrote: > On 14/02/2021 22:47, David Rowley wrote: >> On Sun, 14 Feb 2021 at 13:15, Seamus Abshere >> wrote: >>> The comment from Robert says: (src/backend/optimizer/path/allpaths.c) >>> >>> /* >>>   * If the use of parallel

Re: Why is Postgres only using 8 cores for partitioned count? [Parallel Append]

2021-02-15 Thread Laurenz Albe
On Sun, 2021-02-14 at 22:47 +1300, David Rowley wrote: > On Sun, 14 Feb 2021 at 13:15, Seamus Abshere > > wrote: > > > The comment from Robert says: (src/backend/optimizer/path/allpaths.c) > > /* > > * If the use of parallel append is permitted, always > >

Re: Why is Postgres only using 8 cores for partitioned count? [Parallel Append]

2021-02-14 Thread Gavin Flower
On 14/02/2021 22:47, David Rowley wrote: On Sun, 14 Feb 2021 at 13:15, Seamus Abshere wrote: The comment from Robert says: (src/backend/optimizer/path/allpaths.c) /* * If the use of parallel append is permitted, always request at least *

Re: Why is Postgres only using 8 cores for partitioned count? [Parallel Append]

2021-02-14 Thread Seamus Abshere
Hi David, That is a great suggestion, except my partitions are foreign tables. I wonder if I could figure out a single empty partition that’s just a normal table... but I’m using hash partitioning, so I think I would lose a modulus. Best, Seamus -- Seamus Abshere, SCEA +1 (608) 772-0696

700% faster (was: Re: Why is Postgres only using 8 cores for partitioned count? [Parallel append])

2021-02-14 Thread Seamus Abshere
hi, Apologies for the self-replying, but I made it 700% faster by compiling from source and forcing the parallel_workers from 7 to 96. If this weren't a partitioned table, I could just do `alter table test_3pd_cstore_partitioned set (parallel_workers = 96)` - but that isn't currently

Re: Why is Postgres only using 8 cores for partitioned count? [Parallel Append]

2021-02-14 Thread David Rowley
On Sun, 14 Feb 2021 at 13:15, Seamus Abshere wrote: > The comment from Robert says: (src/backend/optimizer/path/allpaths.c) > > /* > * If the use of parallel append is permitted, always request > at least > * log2(# of children) workers. > > In

Re: Why is Postgres only using 8 cores for partitioned count? [Parallel Append]

2021-02-13 Thread Seamus Abshere
hi, I've traced this back to the formula for Parallel Append workers - log2(partitions). The comment from Robert says: (src/backend/optimizer/path/allpaths.c) /* * If the use of parallel append is permitted, always request at least * log2(# of

Why is Postgres only using 8 cores for partitioned count?

2021-02-13 Thread Seamus Abshere
hi, How can I convince Postgres to use more than 8 cores? I've got an r6gd.16xlarge with 64 vCpus and 512gb RAM running Postgres 13.1 on Ubuntu 20.04. CREATE TABLE tbl ( [...] ) PARTITION BY HASH (address_key); It has 64 partitions - e.g. FOR VALUES WITH (modulus 64, remainder 0) etc.