Re: Should we warn against using too many partitions?

2019-06-12 Thread David Rowley
On Wed, 12 Jun 2019 at 17:49, Amit Langote wrote: > I noticed a typo: > > "...able to handle partition hierarchies up a few thousand partitions" > > s/up/up to/g > > I'm inclined to add one more word though, as: > > "...able to handle partition hierarchies with up to a few thousand partitions" >

Re: Should we warn against using too many partitions?

2019-06-11 Thread Amit Langote
On Wed, Jun 12, 2019 at 5:12 AM David Rowley wrote: > > On Tue, 11 Jun 2019 at 14:53, Amit Langote wrote: > > The latest version looks good to me too. > > Pushed. Thank you all for the reviews. Thanks. I noticed a typo: "...able to handle partition hierarchies up a few thousand partitions"

Re: Should we warn against using too many partitions?

2019-06-11 Thread David Rowley
On Tue, 11 Jun 2019 at 14:53, Amit Langote wrote: > The latest version looks good to me too. Pushed. Thank you all for the reviews. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: Should we warn against using too many partitions?

2019-06-10 Thread Amit Langote
On Tue, Jun 11, 2019 at 11:43 AM Alvaro Herrera wrote: > > On 2019-Jun-11, David Rowley wrote: > > > Changes attached. > > Unreserved +1 to these patches. The latest version looks good to me too. Thanks, Amit

Re: Should we warn against using too many partitions?

2019-06-10 Thread Alvaro Herrera
On 2019-Jun-11, David Rowley wrote: > Changes attached. Unreserved +1 to these patches. -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Should we warn against using too many partitions?

2019-06-10 Thread David Rowley
On Tue, 11 Jun 2019 at 11:15, Justin Pryzby wrote: > > On Mon, Jun 10, 2019 at 06:11:35PM -0400, Alvaro Herrera wrote: > > On 2019-Jun-09, Justin Pryzby wrote: > > > > > I think it should say: > > > > > > |Choosing the target number of partitions into which the table should > > > be > > > |

Re: Should we warn against using too many partitions?

2019-06-10 Thread Justin Pryzby
On Mon, Jun 10, 2019 at 06:11:35PM -0400, Alvaro Herrera wrote: > On 2019-Jun-09, Justin Pryzby wrote: > > > I think it should say: > > > > |Choosing the target number of partitions into which the table should be > > |divided is also a critical decision to make. Not having enough > > I

Re: Should we warn against using too many partitions?

2019-06-10 Thread Alvaro Herrera
On 2019-Jun-09, Justin Pryzby wrote: > I think it should say: > > |Choosing the target number of partitions into which the table should be > |divided is also a critical decision to make. Not having enough I opined elsewhere in the thread that this phrase can be made into more

Re: Should we warn against using too many partitions?

2019-06-10 Thread David Rowley
Thanks for looking at this. On Tue, 11 Jun 2019 at 01:44, Alvaro Herrera wrote: > > part_doc_pg10_v5.patch : > +query planning and execution. The query planner is generally able to > +handle partition hierarchies up a few hundred partition. Planning times > > "up TO a few hundred

Re: Should we warn against using too many partitions?

2019-06-10 Thread Alvaro Herrera
part_doc_pg10_v5.patch : +query planning and execution. The query planner is generally able to +handle partition hierarchies up a few hundred partition. Planning times "up TO a few hundred partition*S*" ? part_doc_master_v5.patch: +Choosing the target number of partitions into

Re: Should we warn against using too many partitions?

2019-06-10 Thread David Rowley
On Mon, 10 Jun 2019 at 20:11, Amit Langote wrote: > > On Sun, Jun 9, 2019 at 5:29 AM David Rowley > > I also split this out into its own paragraph since it's talking about > > something quite different from the previous paragraph. > > Did you miss to split? In v4 patches, I still see this point

Re: Should we warn against using too many partitions?

2019-06-10 Thread Amit Langote
Hi, Thanks for the updated patches. On Sun, Jun 9, 2019 at 5:29 AM David Rowley wrote: > On Fri, 7 Jun 2019 at 19:00, Amit Langote wrote: > > Maybe: > > > > ...Removal of unwanted data is also a factor to consider when > > planning your partitioning strategy as an entire partition can be >

Re: Should we warn against using too many partitions?

2019-06-08 Thread David Rowley
On Sun, 9 Jun 2019 at 17:11, Justin Pryzby wrote: > Sorry, but I think this is still an issue: > > >Choosing the target number of partitions into which the table should be > >divided by is also a critical decision to make. Not having enough > > I think it should say: > > |Choosing

Re: Should we warn against using too many partitions?

2019-06-08 Thread Justin Pryzby
On Sun, Jun 09, 2019 at 05:07:39PM +1200, David Rowley wrote: > On Sun, 9 Jun 2019 at 16:21, Justin Pryzby wrote: > > I meant it should say "into which it should be divided" and not "by which it > > should be divided INTO", which has too many prepositions. This is still an > > issue: > > It now

Re: Should we warn against using too many partitions?

2019-06-08 Thread David Rowley
On Sun, 9 Jun 2019 at 16:21, Justin Pryzby wrote: > I meant it should say "into which it should be divided" and not "by which it > should be divided INTO", which has too many prepositions. This is still an > issue: It now reads "divided by" instead of "divided into". > |these two types of

Re: Should we warn against using too many partitions?

2019-06-08 Thread Justin Pryzby
On Sun, Jun 09, 2019 at 01:15:09PM +1200, David Rowley wrote: > Thanks for having another look. > > On Sat, 8 Jun 2019 at 18:39, Justin Pryzby wrote: > > +to keep exists in that partition then that means having to resort to > > using > > +DELETE instead of removing the partition. > > +

Re: Should we warn against using too many partitions?

2019-06-08 Thread David Rowley
Thanks for having another look. On Sat, 8 Jun 2019 at 18:39, Justin Pryzby wrote: > + > +The choice of how to partition a table should be made carefully as the > +performance of query planning and execution can be negatively affected by > +poorly made design decisions. > > Maybe

Re: Should we warn against using too many partitions?

2019-06-08 Thread David Rowley
Thanks for these suggestions. On Fri, 7 Jun 2019 at 19:00, Amit Langote wrote: > Some rewording suggestions. > > 1. > > +...Removal of unwanted data is also a factor to consider when > +planning your partitioning strategy as an entire partition can be removed > +fairly quickly.

Re: Should we warn against using too many partitions?

2019-06-08 Thread Justin Pryzby
I made another pass, hopefully it's useful and not too much of a pain. diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index cce1618fc1..be2ca3be48 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -4674,6 +4675,88 @@ EXPLAIN SELECT count(*) FROM measurement WHERE

Re: Should we warn against using too many partitions?

2019-06-07 Thread Amit Langote
Hi, Thanks for the updated patches. On Fri, Jun 7, 2019 at 2:34 PM David Rowley wrote: > Anyway comments welcome. If I had a few more minutes to spare I'd > have wrapped OLTP in tags, but out of time for now. Some rewording suggestions. 1. +...Removal of unwanted data is also a

Re: Should we warn against using too many partitions?

2019-06-06 Thread David Rowley
On Fri, 7 Jun 2019 at 03:12, Alvaro Herrera wrote: > I think it'd be worthwhile to mention sub-partitioning. In the attached I did briefly mention about sub-partitioning, however, I didn't feel I had any very wise words to write about it other than it can be useful to split up larger partitions.

Re: Should we warn against using too many partitions?

2019-06-06 Thread David Rowley
On Fri, 7 Jun 2019 at 06:54, Justin Pryzby wrote: > > > >+critical decision to make. Not having enough partitions may mean > > > >that > > > >+indexes remain too large and that data locality remains poor which > > > >could > > > >+result in poor cache hit ratios. However, dividing

Re: Should we warn against using too many partitions?

2019-06-06 Thread David Rowley
On Fri, 7 Jun 2019 at 03:12, Alvaro Herrera wrote: > I think in PG10 something should be mentioned about PK and UNIQUE, so > that people doing their partitioning on that release can think ahead. That seems reasonable, but I feel caution would be required as we don't want to provide any details

Re: Should we warn against using too many partitions?

2019-06-06 Thread Justin Pryzby
On Fri, Jun 07, 2019 at 06:46:59AM +1200, David Rowley wrote: > On Thu, 6 Jun 2019 at 17:29, Justin Pryzby wrote: > > >+ > > >+ > > >+ Declarative Partitioning Best Practices > > >+ > > >+ > > >+The choice of how to partition a table should be considered carefully > > >as > > > >

Re: Should we warn against using too many partitions?

2019-06-06 Thread David Rowley
On Thu, 6 Jun 2019 at 17:29, Justin Pryzby wrote: > >+ > >+ > >+ Declarative Partitioning Best Practices > >+ > >+ > >+The choice of how to partition a table should be considered carefully as > > Either say "How to partition consider should be .." or "The choice should > MADE

Re: Should we warn against using too many partitions?

2019-06-06 Thread Alvaro Herrera
On 2019-Jun-06, David Rowley wrote: > The attached patch is aimed at master. PG11 will need the planner > memory and performance part tweaked and for PG10 I'll do that plus > remove the mention of PRIMARY KEY and UNIQUE constraints on the > partitioned table. I think in PG10 something should be

Re: Should we warn against using too many partitions?

2019-06-05 Thread Amit Langote
Hi, On Thu, Jun 6, 2019 at 1:44 PM David Rowley wrote: > > On Fri, 24 May 2019 at 22:00, David Rowley > wrote: > > I've attached the pg10 and pg11 patches with that updated... and also > > the master one (unchanged) with the hopes that the CF bot picks that > > one. > > I got talking to Andres

Re: Should we warn against using too many partitions?

2019-06-05 Thread Justin Pryzby
I suggest just minor variations on language. On Thu, Jun 06, 2019 at 04:43:48PM +1200, David Rowley wrote: >diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml >index cce1618fc1..ab26630199 100644 >--- a/doc/src/sgml/ddl.sgml >+++ b/doc/src/sgml/ddl.sgml >@@ -4674,6 +4675,76 @@ EXPLAIN

Re: Should we warn against using too many partitions?

2019-06-05 Thread David Rowley
On Fri, 24 May 2019 at 22:00, David Rowley wrote: > I've attached the pg10 and pg11 patches with that updated... and also > the master one (unchanged) with the hopes that the CF bot picks that > one. I got talking to Andres about this at PGCon after a use case of 250k partitions was brought to

Re: Should we warn against using too many partitions?

2019-05-24 Thread David Rowley
On Fri, 24 May 2019 at 17:58, Amit Langote wrote: > + Whether using table inheritance or native partitioning, hierarchies > > Maybe, it would better to use the word "declarative" instead of "native", > if only to be consistent; neighboring paragraphs use "declarative". Thanks for having a

Re: Should we warn against using too many partitions?

2019-05-23 Thread Amit Langote
On 2019/05/24 13:37, David Rowley wrote: > I've attached 3 patches of what I think should go into master, pg11, and pg10. Thanks for the updated patches. In pg11 and pg10 patches, I see this text: + Whether using table inheritance or native partitioning, hierarchies Maybe, it would better

Re: Should we warn against using too many partitions?

2019-05-23 Thread David Rowley
On Fri, 24 May 2019 at 14:04, Amit Langote wrote: > The latest patch on the thread linked from this CF entry (a modified > version of your patch sent by Justin Pryzby) looks good to me. Why not > post it on this thread and link this one to the CF entry? I'm not much of a fan of that patch: +

Re: Should we warn against using too many partitions?

2019-05-23 Thread Amit Langote
Hi David, On 2019/05/23 18:02, David Rowley wrote: > Over on [1] I raised a concern about the lack of any warning in our > documents to inform users that they might not want to use thousands of > partitions. More recently there's [2], also suffering from OOM using > 100 partitions. Perhaps

Should we warn against using too many partitions?

2019-05-23 Thread David Rowley
Over on [1] I raised a concern about the lack of any warning in our documents to inform users that they might not want to use thousands of partitions. More recently there's [2], also suffering from OOM using 100 partitions. Perhaps there's more too this, but the planner using a lot of memory