Over on [1] I raised a concern about the lack of any warning in our
documents to inform users that they might not want to use thousands of
partitions. More recently there's [2], also suffering from OOM using
100 partitions. Perhaps there's more too this, but the planner using
a lot of memory plan
Over on [1] I raised a concern about the lack of any warning in our
documents to inform users that they might not want to use thousands of
partitions. More recently there's [2], also suffering from OOM using
100 partitions. Perhaps there's more too this, but the planner using
a lot of memory plan
Hi David,
On 2019/05/23 18:02, David Rowley wrote:
> Over on [1] I raised a concern about the lack of any warning in our
> documents to inform users that they might not want to use thousands of
> partitions. More recently there's [2], also suffering from OOM using
> 100 partitions. Perhaps there
On Fri, 24 May 2019 at 14:04, Amit Langote
wrote:
> The latest patch on the thread linked from this CF entry (a modified
> version of your patch sent by Justin Pryzby) looks good to me. Why not
> post it on this thread and link this one to the CF entry?
I'm not much of a fan of that patch:
+
On 2019/05/24 13:37, David Rowley wrote:
> I've attached 3 patches of what I think should go into master, pg11, and pg10.
Thanks for the updated patches.
In pg11 and pg10 patches, I see this text:
+ Whether using table inheritance or native partitioning, hierarchies
Maybe, it would better
On Fri, 24 May 2019 at 17:58, Amit Langote
wrote:
> + Whether using table inheritance or native partitioning, hierarchies
>
> Maybe, it would better to use the word "declarative" instead of "native",
> if only to be consistent; neighboring paragraphs use "declarative".
Thanks for having a lo
On Fri, 24 May 2019 at 22:00, David Rowley wrote:
> I've attached the pg10 and pg11 patches with that updated... and also
> the master one (unchanged) with the hopes that the CF bot picks that
> one.
I got talking to Andres about this at PGCon after a use case of 250k
partitions was brought to ou
I suggest just minor variations on language.
On Thu, Jun 06, 2019 at 04:43:48PM +1200, David Rowley wrote:
>diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
>index cce1618fc1..ab26630199 100644
>--- a/doc/src/sgml/ddl.sgml
>+++ b/doc/src/sgml/ddl.sgml
>@@ -4674,6 +4675,76 @@ EXPLAIN SEL
Hi,
On Thu, Jun 6, 2019 at 1:44 PM David Rowley
wrote:
>
> On Fri, 24 May 2019 at 22:00, David Rowley
> wrote:
> > I've attached the pg10 and pg11 patches with that updated... and also
> > the master one (unchanged) with the hopes that the CF bot picks that
> > one.
>
> I got talking to Andres
On 2019-Jun-06, David Rowley wrote:
> The attached patch is aimed at master. PG11 will need the planner
> memory and performance part tweaked and for PG10 I'll do that plus
> remove the mention of PRIMARY KEY and UNIQUE constraints on the
> partitioned table.
I think in PG10 something should be m
On Thu, 6 Jun 2019 at 17:29, Justin Pryzby wrote:
> >+
> >+
> >+ Declarative Partitioning Best Practices
> >+
> >+
> >+The choice of how to partition a table should be considered carefully as
>
> Either say "How to partition consider should be .." or "The choice should
> MADE carefully
On Fri, Jun 07, 2019 at 06:46:59AM +1200, David Rowley wrote:
> On Thu, 6 Jun 2019 at 17:29, Justin Pryzby wrote:
> > >+
> > >+
> > >+ Declarative Partitioning Best Practices
> > >+
> > >+
> > >+The choice of how to partition a table should be considered carefully
> > >as
> >
> > Eithe
On Fri, 7 Jun 2019 at 03:12, Alvaro Herrera wrote:
> I think in PG10 something should be mentioned about PK and UNIQUE, so
> that people doing their partitioning on that release can think ahead.
That seems reasonable, but I feel caution would be required as we
don't want to provide any details ab
On Fri, 7 Jun 2019 at 06:54, Justin Pryzby wrote:
> > > >+critical decision to make. Not having enough partitions may mean
> > > >that
> > > >+indexes remain too large and that data locality remains poor which
> > > >could
> > > >+result in poor cache hit ratios. However, dividing
On Fri, 7 Jun 2019 at 03:12, Alvaro Herrera wrote:
> I think it'd be worthwhile to mention sub-partitioning.
In the attached I did briefly mention about sub-partitioning, however,
I didn't feel I had any very wise words to write about it other than
it can be useful to split up larger partitions.
Hi,
Thanks for the updated patches.
On Fri, Jun 7, 2019 at 2:34 PM David Rowley
wrote:
> Anyway comments welcome. If I had a few more minutes to spare I'd
> have wrapped OLTP in tags, but out of time for now.
Some rewording suggestions.
1.
+...Removal of unwanted data is also a fact
I made another pass, hopefully it's useful and not too much of a pain.
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index cce1618fc1..be2ca3be48 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -4674,6 +4675,88 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdat
Thanks for these suggestions.
On Fri, 7 Jun 2019 at 19:00, Amit Langote wrote:
> Some rewording suggestions.
>
> 1.
>
> +...Removal of unwanted data is also a factor to consider when
> +planning your partitioning strategy as an entire partition can be removed
> +fairly quickly. H
Thanks for having another look.
On Sat, 8 Jun 2019 at 18:39, Justin Pryzby wrote:
> +
> +The choice of how to partition a table should be made carefully as the
> +performance of query planning and execution can be negatively affected by
> +poorly made design decisions.
>
> Maybe ju
On Sun, Jun 09, 2019 at 01:15:09PM +1200, David Rowley wrote:
> Thanks for having another look.
>
> On Sat, 8 Jun 2019 at 18:39, Justin Pryzby wrote:
> > +to keep exists in that partition then that means having to resort to
> > using
> > +DELETE instead of removing the partition.
> > +
On Sun, 9 Jun 2019 at 16:21, Justin Pryzby wrote:
> I meant it should say "into which it should be divided" and not "by which it
> should be divided INTO", which has too many prepositions. This is still an
> issue:
It now reads "divided by" instead of "divided into".
> |these two types of w
On Sun, Jun 09, 2019 at 05:07:39PM +1200, David Rowley wrote:
> On Sun, 9 Jun 2019 at 16:21, Justin Pryzby wrote:
> > I meant it should say "into which it should be divided" and not "by which it
> > should be divided INTO", which has too many prepositions. This is still an
> > issue:
>
> It now
On Sun, 9 Jun 2019 at 17:11, Justin Pryzby wrote:
> Sorry, but I think this is still an issue:
>
> >Choosing the target number of partitions into which the table should be
> >divided by is also a critical decision to make. Not having enough
>
> I think it should say:
>
> |Choosing the
Hi,
Thanks for the updated patches.
On Sun, Jun 9, 2019 at 5:29 AM David Rowley
wrote:
> On Fri, 7 Jun 2019 at 19:00, Amit Langote wrote:
> > Maybe:
> >
> > ...Removal of unwanted data is also a factor to consider when
> > planning your partitioning strategy as an entire partition can be
>
On Mon, 10 Jun 2019 at 20:11, Amit Langote wrote:
>
> On Sun, Jun 9, 2019 at 5:29 AM David Rowley
> > I also split this out into its own paragraph since it's talking about
> > something quite different from the previous paragraph.
>
> Did you miss to split? In v4 patches, I still see this point
>
part_doc_pg10_v5.patch :
+query planning and execution. The query planner is generally able to
+handle partition hierarchies up a few hundred partition. Planning times
"up TO a few hundred partition*S*" ?
part_doc_master_v5.patch:
+Choosing the target number of partitions into whic
Thanks for looking at this.
On Tue, 11 Jun 2019 at 01:44, Alvaro Herrera wrote:
>
> part_doc_pg10_v5.patch :
> +query planning and execution. The query planner is generally able to
> +handle partition hierarchies up a few hundred partition. Planning times
>
> "up TO a few hundred partit
On 2019-Jun-09, Justin Pryzby wrote:
> I think it should say:
>
> |Choosing the target number of partitions into which the table should be
> |divided is also a critical decision to make. Not having enough
I opined elsewhere in the thread that this phrase can be made into more
straightfo
On Mon, Jun 10, 2019 at 06:11:35PM -0400, Alvaro Herrera wrote:
> On 2019-Jun-09, Justin Pryzby wrote:
>
> > I think it should say:
> >
> > |Choosing the target number of partitions into which the table should be
> > |divided is also a critical decision to make. Not having enough
>
> I
On Tue, 11 Jun 2019 at 11:15, Justin Pryzby wrote:
>
> On Mon, Jun 10, 2019 at 06:11:35PM -0400, Alvaro Herrera wrote:
> > On 2019-Jun-09, Justin Pryzby wrote:
> >
> > > I think it should say:
> > >
> > > |Choosing the target number of partitions into which the table should
> > > be
> > > |
On 2019-Jun-11, David Rowley wrote:
> Changes attached.
Unreserved +1 to these patches.
--
Álvaro Herrerahttps://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On Tue, Jun 11, 2019 at 11:43 AM Alvaro Herrera
wrote:
>
> On 2019-Jun-11, David Rowley wrote:
>
> > Changes attached.
>
> Unreserved +1 to these patches.
The latest version looks good to me too.
Thanks,
Amit
On Tue, 11 Jun 2019 at 14:53, Amit Langote wrote:
> The latest version looks good to me too.
Pushed. Thank you all for the reviews.
--
David Rowley http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
On Wed, Jun 12, 2019 at 5:12 AM David Rowley
wrote:
>
> On Tue, 11 Jun 2019 at 14:53, Amit Langote wrote:
> > The latest version looks good to me too.
>
> Pushed. Thank you all for the reviews.
Thanks.
I noticed a typo:
"...able to handle partition hierarchies up a few thousand partitions"
s
On Wed, 12 Jun 2019 at 17:49, Amit Langote wrote:
> I noticed a typo:
>
> "...able to handle partition hierarchies up a few thousand partitions"
>
> s/up/up to/g
>
> I'm inclined to add one more word though, as:
>
> "...able to handle partition hierarchies with up to a few thousand partitions"
>
>
Hi David,
On 2019/05/23 18:02, David Rowley wrote:
> Over on [1] I raised a concern about the lack of any warning in our
> documents to inform users that they might not want to use thousands of
> partitions. More recently there's [2], also suffering from OOM using
> 100 partitions. Perhaps there
On Fri, 24 May 2019 at 14:04, Amit Langote
wrote:
> The latest patch on the thread linked from this CF entry (a modified
> version of your patch sent by Justin Pryzby) looks good to me. Why not
> post it on this thread and link this one to the CF entry?
I'm not much of a fan of that patch:
+
On 2019/05/24 13:37, David Rowley wrote:
> I've attached 3 patches of what I think should go into master, pg11, and pg10.
Thanks for the updated patches.
In pg11 and pg10 patches, I see this text:
+ Whether using table inheritance or native partitioning, hierarchies
Maybe, it would better
On Fri, 24 May 2019 at 17:58, Amit Langote
wrote:
> + Whether using table inheritance or native partitioning, hierarchies
>
> Maybe, it would better to use the word "declarative" instead of "native",
> if only to be consistent; neighboring paragraphs use "declarative".
Thanks for having a lo
On Fri, 24 May 2019 at 22:00, David Rowley wrote:
> I've attached the pg10 and pg11 patches with that updated... and also
> the master one (unchanged) with the hopes that the CF bot picks that
> one.
I got talking to Andres about this at PGCon after a use case of 250k
partitions was brought to ou
I suggest just minor variations on language.
On Thu, Jun 06, 2019 at 04:43:48PM +1200, David Rowley wrote:
>diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
>index cce1618fc1..ab26630199 100644
>--- a/doc/src/sgml/ddl.sgml
>+++ b/doc/src/sgml/ddl.sgml
>@@ -4674,6 +4675,76 @@ EXPLAIN SEL
Hi,
On Thu, Jun 6, 2019 at 1:44 PM David Rowley
wrote:
>
> On Fri, 24 May 2019 at 22:00, David Rowley
> wrote:
> > I've attached the pg10 and pg11 patches with that updated... and also
> > the master one (unchanged) with the hopes that the CF bot picks that
> > one.
>
> I got talking to Andres
On 2019-Jun-06, David Rowley wrote:
> The attached patch is aimed at master. PG11 will need the planner
> memory and performance part tweaked and for PG10 I'll do that plus
> remove the mention of PRIMARY KEY and UNIQUE constraints on the
> partitioned table.
I think in PG10 something should be m
On Thu, 6 Jun 2019 at 17:29, Justin Pryzby wrote:
> >+
> >+
> >+ Declarative Partitioning Best Practices
> >+
> >+
> >+The choice of how to partition a table should be considered carefully as
>
> Either say "How to partition consider should be .." or "The choice should
> MADE carefully
On Fri, Jun 07, 2019 at 06:46:59AM +1200, David Rowley wrote:
> On Thu, 6 Jun 2019 at 17:29, Justin Pryzby wrote:
> > >+
> > >+
> > >+ Declarative Partitioning Best Practices
> > >+
> > >+
> > >+The choice of how to partition a table should be considered carefully
> > >as
> >
> > Eithe
On Fri, 7 Jun 2019 at 03:12, Alvaro Herrera wrote:
> I think in PG10 something should be mentioned about PK and UNIQUE, so
> that people doing their partitioning on that release can think ahead.
That seems reasonable, but I feel caution would be required as we
don't want to provide any details ab
On Fri, 7 Jun 2019 at 06:54, Justin Pryzby wrote:
> > > >+critical decision to make. Not having enough partitions may mean
> > > >that
> > > >+indexes remain too large and that data locality remains poor which
> > > >could
> > > >+result in poor cache hit ratios. However, dividing
On Fri, 7 Jun 2019 at 03:12, Alvaro Herrera wrote:
> I think it'd be worthwhile to mention sub-partitioning.
In the attached I did briefly mention about sub-partitioning, however,
I didn't feel I had any very wise words to write about it other than
it can be useful to split up larger partitions.
Hi,
Thanks for the updated patches.
On Fri, Jun 7, 2019 at 2:34 PM David Rowley
wrote:
> Anyway comments welcome. If I had a few more minutes to spare I'd
> have wrapped OLTP in tags, but out of time for now.
Some rewording suggestions.
1.
+...Removal of unwanted data is also a fact
I made another pass, hopefully it's useful and not too much of a pain.
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index cce1618fc1..be2ca3be48 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -4674,6 +4675,88 @@ EXPLAIN SELECT count(*) FROM measurement WHERE logdat
Thanks for these suggestions.
On Fri, 7 Jun 2019 at 19:00, Amit Langote wrote:
> Some rewording suggestions.
>
> 1.
>
> +...Removal of unwanted data is also a factor to consider when
> +planning your partitioning strategy as an entire partition can be removed
> +fairly quickly. H
Thanks for having another look.
On Sat, 8 Jun 2019 at 18:39, Justin Pryzby wrote:
> +
> +The choice of how to partition a table should be made carefully as the
> +performance of query planning and execution can be negatively affected by
> +poorly made design decisions.
>
> Maybe ju
On Sun, Jun 09, 2019 at 01:15:09PM +1200, David Rowley wrote:
> Thanks for having another look.
>
> On Sat, 8 Jun 2019 at 18:39, Justin Pryzby wrote:
> > +to keep exists in that partition then that means having to resort to
> > using
> > +DELETE instead of removing the partition.
> > +
On Sun, 9 Jun 2019 at 16:21, Justin Pryzby wrote:
> I meant it should say "into which it should be divided" and not "by which it
> should be divided INTO", which has too many prepositions. This is still an
> issue:
It now reads "divided by" instead of "divided into".
> |these two types of w
On Sun, Jun 09, 2019 at 05:07:39PM +1200, David Rowley wrote:
> On Sun, 9 Jun 2019 at 16:21, Justin Pryzby wrote:
> > I meant it should say "into which it should be divided" and not "by which it
> > should be divided INTO", which has too many prepositions. This is still an
> > issue:
>
> It now
On Sun, 9 Jun 2019 at 17:11, Justin Pryzby wrote:
> Sorry, but I think this is still an issue:
>
> >Choosing the target number of partitions into which the table should be
> >divided by is also a critical decision to make. Not having enough
>
> I think it should say:
>
> |Choosing the
Hi,
Thanks for the updated patches.
On Sun, Jun 9, 2019 at 5:29 AM David Rowley
wrote:
> On Fri, 7 Jun 2019 at 19:00, Amit Langote wrote:
> > Maybe:
> >
> > ...Removal of unwanted data is also a factor to consider when
> > planning your partitioning strategy as an entire partition can be
>
On Mon, 10 Jun 2019 at 20:11, Amit Langote wrote:
>
> On Sun, Jun 9, 2019 at 5:29 AM David Rowley
> > I also split this out into its own paragraph since it's talking about
> > something quite different from the previous paragraph.
>
> Did you miss to split? In v4 patches, I still see this point
>
part_doc_pg10_v5.patch :
+query planning and execution. The query planner is generally able to
+handle partition hierarchies up a few hundred partition. Planning times
"up TO a few hundred partition*S*" ?
part_doc_master_v5.patch:
+Choosing the target number of partitions into whic
Thanks for looking at this.
On Tue, 11 Jun 2019 at 01:44, Alvaro Herrera wrote:
>
> part_doc_pg10_v5.patch :
> +query planning and execution. The query planner is generally able to
> +handle partition hierarchies up a few hundred partition. Planning times
>
> "up TO a few hundred partit
On 2019-Jun-09, Justin Pryzby wrote:
> I think it should say:
>
> |Choosing the target number of partitions into which the table should be
> |divided is also a critical decision to make. Not having enough
I opined elsewhere in the thread that this phrase can be made into more
straightfo
On Mon, Jun 10, 2019 at 06:11:35PM -0400, Alvaro Herrera wrote:
> On 2019-Jun-09, Justin Pryzby wrote:
>
> > I think it should say:
> >
> > |Choosing the target number of partitions into which the table should be
> > |divided is also a critical decision to make. Not having enough
>
> I
On Tue, 11 Jun 2019 at 11:15, Justin Pryzby wrote:
>
> On Mon, Jun 10, 2019 at 06:11:35PM -0400, Alvaro Herrera wrote:
> > On 2019-Jun-09, Justin Pryzby wrote:
> >
> > > I think it should say:
> > >
> > > |Choosing the target number of partitions into which the table should
> > > be
> > > |
On 2019-Jun-11, David Rowley wrote:
> Changes attached.
Unreserved +1 to these patches.
--
Álvaro Herrerahttps://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On Tue, Jun 11, 2019 at 11:43 AM Alvaro Herrera
wrote:
>
> On 2019-Jun-11, David Rowley wrote:
>
> > Changes attached.
>
> Unreserved +1 to these patches.
The latest version looks good to me too.
Thanks,
Amit
On Tue, 11 Jun 2019 at 14:53, Amit Langote wrote:
> The latest version looks good to me too.
Pushed. Thank you all for the reviews.
--
David Rowley http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
On Wed, Jun 12, 2019 at 5:12 AM David Rowley
wrote:
>
> On Tue, 11 Jun 2019 at 14:53, Amit Langote wrote:
> > The latest version looks good to me too.
>
> Pushed. Thank you all for the reviews.
Thanks.
I noticed a typo:
"...able to handle partition hierarchies up a few thousand partitions"
s
On Wed, 12 Jun 2019 at 17:49, Amit Langote wrote:
> I noticed a typo:
>
> "...able to handle partition hierarchies up a few thousand partitions"
>
> s/up/up to/g
>
> I'm inclined to add one more word though, as:
>
> "...able to handle partition hierarchies with up to a few thousand partitions"
>
>
68 matches
Mail list logo