Re: Partitioning: issues/ideas (Was: Re: [HACKERS] On partitioning)

2015-01-27 Thread Amit Langote
On 27-01-2015 AM 05:46, Jim Nasby wrote: > On 1/25/15 7:42 PM, Amit Langote wrote: >> On 21-01-2015 PM 07:26, Amit Langote wrote: >>> Ok, I will limit myself to focusing on following things at the moment: >>> >>> * Provide syntax in CREATE TABLE to declare partition key >> >> While working on this,

Re: Partitioning: issues/ideas (Was: Re: [HACKERS] On partitioning)

2015-01-26 Thread Jim Nasby
On 1/25/15 7:42 PM, Amit Langote wrote: On 21-01-2015 PM 07:26, Amit Langote wrote: Ok, I will limit myself to focusing on following things at the moment: * Provide syntax in CREATE TABLE to declare partition key While working on this, I stumbled upon the question of how we deal with any inde

Re: Partitioning: issues/ideas (Was: Re: [HACKERS] On partitioning)

2015-01-25 Thread Amit Langote
On 21-01-2015 PM 07:26, Amit Langote wrote: > Ok, I will limit myself to focusing on following things at the moment: > > * Provide syntax in CREATE TABLE to declare partition key While working on this, I stumbled upon the question of how we deal with any index definitions following from constrain

Re: Partitioning: issues/ideas (Was: Re: [HACKERS] On partitioning)

2015-01-21 Thread Amit Langote
On 21-01-2015 AM 01:42, Robert Haas wrote: > On Mon, Jan 19, 2015 at 8:48 PM, Amit Langote > wrote: Specifically, do we regard a partitions as pg_inherits children of its partitioning parent? >>> >>> I don't think this is totally an all-or-nothing decision. I think >>> everyone is agree

Re: Partitioning: issues/ideas (Was: Re: [HACKERS] On partitioning)

2015-01-20 Thread Robert Haas
On Mon, Jan 19, 2015 at 8:48 PM, Amit Langote wrote: >>> Specifically, do we regard a partitions as pg_inherits children of its >>> partitioning parent? >> >> I don't think this is totally an all-or-nothing decision. I think >> everyone is agreed that we need to not break things that work today -

Re: Partitioning: issues/ideas (Was: Re: [HACKERS] On partitioning)

2015-01-19 Thread Amit Langote
On 20-01-2015 AM 10:48, Amit Langote wrote: > On 17-01-2015 AM 02:34, Robert Haas wrote: >> On Wed, Jan 14, 2015 at 9:07 PM, Amit Langote >> wrote: >>> * It is desirable to treat partitions as pg_class relations with perhaps >>> a new relkind(s). We may want to choose an implementation where only

Re: Partitioning: issues/ideas (Was: Re: [HACKERS] On partitioning)

2015-01-19 Thread Amit Langote
On 17-01-2015 AM 02:34, Robert Haas wrote: > On Wed, Jan 14, 2015 at 9:07 PM, Amit Langote > wrote: >> * It has been repeatedly pointed out that we may want to decouple >> partitioning from inheritance because implementing partitioning as an >> extension of inheritance mechanism means that we have

Re: Partitioning: issues/ideas (Was: Re: [HACKERS] On partitioning)

2015-01-18 Thread Amit Langote
On 19-01-2015 PM 12:37, Ashutosh Bapat wrote: > On Fri, Jan 16, 2015 at 11:04 PM, Robert Haas wrote: > >> On Wed, Jan 14, 2015 at 9:07 PM, Amit Langote >> wrote: >> >>> I wonder if we could add a clause like DISTRIBUTED BY to complement >>> PARTITION ON that represents a hash distributed/partiti

Re: Partitioning: issues/ideas (Was: Re: [HACKERS] On partitioning)

2015-01-18 Thread Ashutosh Bapat
On Fri, Jan 16, 2015 at 11:04 PM, Robert Haas wrote: > On Wed, Jan 14, 2015 at 9:07 PM, Amit Langote > wrote: > > * It has been repeatedly pointed out that we may want to decouple > > partitioning from inheritance because implementing partitioning as an > > extension of inheritance mechanism mea

Re: Partitioning: issues/ideas (Was: Re: [HACKERS] On partitioning)

2015-01-16 Thread Robert Haas
On Wed, Jan 14, 2015 at 9:07 PM, Amit Langote wrote: > * It has been repeatedly pointed out that we may want to decouple > partitioning from inheritance because implementing partitioning as an > extension of inheritance mechanism means that we have to keep all the > existing semantics which might

Partitioning: issues/ideas (Was: Re: [HACKERS] On partitioning)

2015-01-14 Thread Amit Langote
On 06-01-2015 PM 03:40, Amit Langote wrote: > > I agree that while we are discussing these points, we could also be > discussing how we solve problems of existing partitioning implementation > using whatever the above things end up being. Proposed approaches to > solve those problems might be usef

Re: [HACKERS] On partitioning

2015-01-05 Thread Amit Langote
On 18-12-2014 AM 04:52, Robert Haas wrote: > On Wed, Dec 17, 2014 at 1:53 PM, Josh Berkus wrote: >> >> Sure. But there's a big difference between "we're going to take these >> steps and that problem will be fixable eventually" and "we're going to >> retain features of the current partitioning sys

Re: [HACKERS] On partitioning

2014-12-17 Thread Robert Haas
On Wed, Dec 17, 2014 at 1:53 PM, Josh Berkus wrote: > On 12/16/2014 07:35 PM, Robert Haas wrote: >> On Tue, Dec 16, 2014 at 9:01 PM, Josh Berkus wrote: >>> Anyway, what I'm saying is that I personally regard the inability to >>> handle even moderately complex expressions a major failing of our >>

Re: [HACKERS] On partitioning

2014-12-17 Thread Josh Berkus
On 12/17/2014 11:19 AM, Heikki Linnakangas wrote: > On 12/17/2014 08:53 PM, Josh Berkus wrote: >> Last week, I wrote a longish email listing out the common problems users >> have with our current partitioning as a way of benchmarking the plan for >> new partitioning. While some people responded to

Re: [HACKERS] On partitioning

2014-12-17 Thread Heikki Linnakangas
On 12/17/2014 08:53 PM, Josh Berkus wrote: Last week, I wrote a longish email listing out the common problems users have with our current partitioning as a way of benchmarking the plan for new partitioning. While some people responded to that post, absolutely nobody discussed the list of issues

Re: [HACKERS] On partitioning

2014-12-17 Thread Josh Berkus
On 12/16/2014 07:35 PM, Robert Haas wrote: > On Tue, Dec 16, 2014 at 9:01 PM, Josh Berkus wrote: >> Anyway, what I'm saying is that I personally regard the inability to >> handle even moderately complex expressions a major failing of our >> existing partitioning scheme (possibly its worst single f

Re: [HACKERS] On partitioning

2014-12-16 Thread Robert Haas
On Tue, Dec 16, 2014 at 9:01 PM, Josh Berkus wrote: > On 12/16/2014 05:52 PM, Robert Haas wrote: >> But in a more complicated case where the value there isn't known until >> runtime, yeah, it scans everything. I'm not sure what the best way to >> fix that is. If the partition bounds were stored

Re: [HACKERS] On partitioning

2014-12-16 Thread Josh Berkus
On 12/16/2014 05:52 PM, Robert Haas wrote: > But in a more complicated case where the value there isn't known until > runtime, yeah, it scans everything. I'm not sure what the best way to > fix that is. If the partition bounds were stored in a structured way, > as we've been discussing, then the

Re: [HACKERS] On partitioning

2014-12-16 Thread Robert Haas
On Tue, Dec 16, 2014 at 1:45 PM, Josh Berkus wrote: > Yes, I wasn't saying that expressions should be used when *creating* the > partitions, which strikes me as a bad idea for several reasons. > Expressions should be usable when SELECTing data from the partitions. > Right now, they aren't, because

Re: [HACKERS] On partitioning

2014-12-16 Thread Amit Langote
On 17-12-2014 AM 12:28, Claudio Freire wrote: > On Tue, Dec 16, 2014 at 12:15 PM, Robert Haas wrote: >> I'm not really sure what you are getting here. An "otherwise-good >> expression" basically means a constant. Index expressions have to be >> things that always produce the same result given th

Re: [HACKERS] On partitioning

2014-12-16 Thread Amit Langote
On 17-12-2014 AM 12:15, Robert Haas wrote: > On Mon, Dec 15, 2014 at 6:55 PM, Amit Langote > wrote: >> Robert wrote: >>> I would expect that to fail, just as it would fail if you tried to >>> build an index using a volatile expression. >> >> Oops, wrong example, sorry. In case of an otherwise good

Re: [HACKERS] On partitioning

2014-12-16 Thread Josh Berkus
On 12/15/2014 10:55 AM, Robert Haas wrote: >> This means if a user puts arbitrary expressions in a partition definition, >> say, >> > >> > ... FOR VALUES extract(month from current_date) TO extract(month from >> > current_date + interval '3 months'), >> > >> > we make sure that those expressions

Re: [HACKERS] On partitioning

2014-12-16 Thread Claudio Freire
On Tue, Dec 16, 2014 at 12:15 PM, Robert Haas wrote: > wrote: >> Robert wrote: >>> On Sun, Dec 14, 2014 at 9:12 PM, Amit Langote >>> wrote: >>> > This means if a user puts arbitrary expressions in a partition >>> > definition, say, >>> > >>> > ... FOR VALUES extract(month from current_date) TO

Re: [HACKERS] On partitioning

2014-12-16 Thread Robert Haas
On Mon, Dec 15, 2014 at 6:55 PM, Amit Langote wrote: > Robert wrote: >> On Sun, Dec 14, 2014 at 9:12 PM, Amit Langote >> wrote: >> > This means if a user puts arbitrary expressions in a partition definition, >> > say, >> > >> > ... FOR VALUES extract(month from current_date) TO extract(month fr

Re: [HACKERS] On partitioning

2014-12-15 Thread Amit Langote
Robert wrote: > On Sun, Dec 14, 2014 at 9:12 PM, Amit Langote > wrote: > > This means if a user puts arbitrary expressions in a partition definition, > > say, > > > > ... FOR VALUES extract(month from current_date) TO extract(month from > current_date + interval '3 months'), > > > > we make sur

Re: [HACKERS] On partitioning

2014-12-15 Thread Robert Haas
On Sun, Dec 14, 2014 at 9:12 PM, Amit Langote wrote: > This means if a user puts arbitrary expressions in a partition definition, > say, > > ... FOR VALUES extract(month from current_date) TO extract(month from > current_date + interval '3 months'), > > we make sure that those expressions are p

Re: [HACKERS] On partitioning

2014-12-15 Thread Claudio Freire
On Mon, Dec 15, 2014 at 8:09 AM, José Luis Tallón wrote: > On 12/15/2014 07:42 AM, Claudio Freire wrote: >> >> [snip] > > >> If you do that, you start with empty partitions, and each insert updates >> the BRIN tuple. Avoiding concurrency loss in this case would be tricky, but >> in theory this cou

Re: [HACKERS] On partitioning

2014-12-15 Thread José Luis Tallón
On 12/15/2014 07:42 AM, Claudio Freire wrote: [snip] If you do that, you start with empty partitions, and each insert updates the BRIN tuple. Avoiding concurrency loss in this case would be tricky, but in theory this could allow very general partition exclusion. In fact it could even work wi

Re: [HACKERS] On partitioning

2014-12-15 Thread Amit Langote
Claudio Freire wrote: > On Sun, Dec 14, 2014 at 11:12 PM, Amit Langote > wrote: > >> On egress you need some direct way to compare the scan quals with the > >> partitioning values. I would imagine this to be similar to how scan > >> quals are compared to the values stored in a BRIN index: each s

Re: [HACKERS] On partitioning

2014-12-14 Thread Claudio Freire
On Sun, Dec 14, 2014 at 11:12 PM, Amit Langote wrote: >> On egress you need some direct way to compare the scan quals with the >> partitioning values. I would imagine this to be similar to how scan >> quals are compared to the values stored in a BRIN index: each scan qual >> has a corresponding o

Re: [HACKERS] On partitioning

2014-12-14 Thread Amit Langote
Alvaro wrote: > Claudio Freire wrote: > > > Fair enough, but that's not the same as not requiring easy proofs. The > > planner might not the one doing the proofs, but you still need proofs. > > > > Even if the proving method is hardcoded into the partitioning method, > > as in the case of list or

Re: [HACKERS] On partitioning

2014-12-13 Thread Amit Langote
On Sun, Dec 14, 2014 at 1:40 AM, José Luis Tallón wrote: > On 12/12/2014 05:43 AM, Amit Langote wrote: > > Amit: mind if I add the DB2 syntax for partitioning to the wiki, too? > > This might as well help with deciding the final form of partitioning > (and define the first implementation bound

Re: [HACKERS] On partitioning

2014-12-13 Thread Amit Langote
On Sun, Dec 14, 2014 at 1:57 AM, José Luis Tallón wrote: > On 12/13/2014 03:09 AM, Alvaro Herrera wrote: >> >> [snip] >> Arbitrary SQL expressions (including functions) are not the thing to use >> for partitioning -- at least that's how I understand this whole >> discussion. I don't think you wan

Re: [HACKERS] On partitioning

2014-12-13 Thread José Luis Tallón
On 12/13/2014 05:57 PM, José Luis Tallón wrote: On 12/13/2014 03:09 AM, Alvaro Herrera wrote: [snip] Arbitrary SQL expressions (including functions) are not the thing to use for partitioning -- at least that's how I understand this whole discussion. I don't think you want to do "proofs" as such

Re: [HACKERS] On partitioning

2014-12-13 Thread David Fetter
On Fri, Dec 12, 2014 at 09:03:12AM -0500, Robert Haas wrote: > Yeah, range and list partition definitions are very similar, but > hash partition definitions are a different kettle of fish. I don't > think we really need hash partitioning for anything right away - > it's pretty useless unless you'

Re: [HACKERS] On partitioning

2014-12-13 Thread Jim Nasby
On 12/12/14, 3:48 PM, Robert Haas wrote: On Fri, Dec 12, 2014 at 4:28 PM, Jim Nasby wrote: Sure. Mind you, I'm not proposing that the syntax I just mooted is actually for the best. What I'm saying is that we need to talk about it. Frankly, if we're going to require users to explicitly defin

Re: [HACKERS] On partitioning

2014-12-13 Thread José Luis Tallón
On 12/13/2014 03:09 AM, Alvaro Herrera wrote: [snip] Arbitrary SQL expressions (including functions) are not the thing to use for partitioning -- at least that's how I understand this whole discussion. I don't think you want to do "proofs" as such -- they are expensive. Yup. Plus, it looks lik

Re: [HACKERS] On partitioning

2014-12-13 Thread José Luis Tallón
On 12/12/2014 05:43 AM, Amit Langote wrote: [snip] In case of what we would have called a 'LIST' partition, this could look like ... FOR VALUES (val1, val2, val3, ...) Assuming we only support partition key to contain only one column in such a case. Hmmm…. [...] PARTITION BY LIST(col1 [, co

Re: [HACKERS] On partitioning

2014-12-12 Thread Claudio Freire
El 12/12/2014 23:09, "Alvaro Herrera" escribió: > > Claudio Freire wrote: > > > Fair enough, but that's not the same as not requiring easy proofs. The > > planner might not the one doing the proofs, but you still need proofs. > > > > Even if the proving method is hardcoded into the partitioning me

Re: [HACKERS] On partitioning

2014-12-12 Thread Alvaro Herrera
Claudio Freire wrote: > Fair enough, but that's not the same as not requiring easy proofs. The > planner might not the one doing the proofs, but you still need proofs. > > Even if the proving method is hardcoded into the partitioning method, > as in the case of list or range partitioning, it's st

Re: [HACKERS] On partitioning

2014-12-12 Thread Claudio Freire
On Fri, Dec 12, 2014 at 7:40 PM, Josh Berkus wrote: > On 12/12/2014 02:10 PM, Tom Lane wrote: >> Actually, I'm not sure that's what we want. I thought what we really >> wanted here was to postpone partition-routing decisions to runtime, >> so that the behavior would be efficient whether or not th

Re: [HACKERS] On partitioning

2014-12-12 Thread Josh Berkus
On 12/12/2014 02:10 PM, Tom Lane wrote: > Actually, I'm not sure that's what we want. I thought what we really > wanted here was to postpone partition-routing decisions to runtime, > so that the behavior would be efficient whether or not the decision > could be predetermined at plan time. > > Thi

Re: [HACKERS] On partitioning

2014-12-12 Thread Claudio Freire
On Fri, Dec 12, 2014 at 7:10 PM, Tom Lane wrote: > Claudio Freire writes: >> On Fri, Dec 12, 2014 at 6:48 PM, Robert Haas wrote: >>> I have very little idea what the API you're imagining would actually >>> look like from this description, but it sounds like a terrible idea. >>> We don't want to

Re: [HACKERS] On partitioning

2014-12-12 Thread Tom Lane
Claudio Freire writes: > On Fri, Dec 12, 2014 at 6:48 PM, Robert Haas wrote: >> I have very little idea what the API you're imagining would actually >> look like from this description, but it sounds like a terrible idea. >> We don't want to make this infinitely general. We need a *fast* way >> t

Re: [HACKERS] On partitioning

2014-12-12 Thread Claudio Freire
On Fri, Dec 12, 2014 at 6:48 PM, Robert Haas wrote: > On Fri, Dec 12, 2014 at 4:28 PM, Jim Nasby wrote: >>> Sure. Mind you, I'm not proposing that the syntax I just mooted is >>> actually for the best. What I'm saying is that we need to talk about >>> it. >> >> Frankly, if we're going to requir

Re: [HACKERS] On partitioning

2014-12-12 Thread Robert Haas
On Fri, Dec 12, 2014 at 4:28 PM, Jim Nasby wrote: >> Sure. Mind you, I'm not proposing that the syntax I just mooted is >> actually for the best. What I'm saying is that we need to talk about >> it. > > Frankly, if we're going to require users to explicitly define each partition > then I think t

Re: [HACKERS] On partitioning

2014-12-12 Thread Jim Nasby
On 12/12/14, 8:03 AM, Robert Haas wrote: On Thu, Dec 11, 2014 at 11:43 PM, Amit Langote wrote: >In case of what we would have called a 'LIST' partition, this could look like > >... FOR VALUES (val1, val2, val3, ...) > >Assuming we only support partition key to contain only one column in such a

Re: [HACKERS] On partitioning

2014-12-12 Thread Robert Haas
On Thu, Dec 11, 2014 at 11:43 PM, Amit Langote wrote: > In case of what we would have called a 'LIST' partition, this could look like > > ... FOR VALUES (val1, val2, val3, ...) > > Assuming we only support partition key to contain only one column in such a > case. > > In case of what we would hav

Re: [HACKERS] On partitioning

2014-12-11 Thread Amit Kapila
On Thu, Dec 11, 2014 at 8:42 PM, Robert Haas wrote: > > On Thu, Dec 11, 2014 at 12:00 AM, Amit Kapila wrote: > > Yeah either this way or what Josh has suggested upthread, the main > > point was that if at all we want to support multi-column list partitioning > > then we need to have slightly diff

Re: [HACKERS] On partitioning

2014-12-11 Thread Amit Langote
> -Original Message- > From: Robert Haas [mailto:robertmh...@gmail.com] > On Thu, Dec 11, 2014 at 12:00 AM, Amit Kapila > wrote: > > Yeah either this way or what Josh has suggested upthread, the main > > point was that if at all we want to support multi-column list partitioning > > then w

Re: [HACKERS] On partitioning

2014-12-11 Thread Robert Haas
On Thu, Dec 11, 2014 at 12:00 AM, Amit Kapila wrote: > Yeah either this way or what Josh has suggested upthread, the main > point was that if at all we want to support multi-column list partitioning > then we need to have slightly different syntax, however I feel that we > can leave multi-column l

Re: [HACKERS] On partitioning

2014-12-10 Thread Amit Kapila
On Wed, Dec 10, 2014 at 11:51 PM, Robert Haas wrote: > > On Mon, Dec 8, 2014 at 10:59 PM, Amit Kapila wrote: > > Yeah and also how would user specify the values, as an example > > assume that table is partitioned on monthly_salary, so partition > > definition would look: > > > > PARTITION BY LIST

Re: [HACKERS] On partitioning

2014-12-10 Thread Amit Kapila
On Wed, Dec 10, 2014 at 7:52 PM, Alvaro Herrera wrote: > > Amit Langote wrote: > > > On Wed, Dec 10, 2014 at 12:46 PM, Amit Kapila wrote: > > > On Tue, Dec 9, 2014 at 7:21 PM, Alvaro Herrera < alvhe...@2ndquadrant.com> > > > wrote: > > > >> FWIW in my original proposal I was rejecting some things

Re: [HACKERS] On partitioning

2014-12-10 Thread Robert Haas
On Wed, Dec 10, 2014 at 7:25 PM, Amit Langote wrote: > In heap_create(), do we create storage for a top level partitioned table > (say, RELKIND_PARTITIONED_TABLE)? How about a partition that is further > sub-partitioned? We might allocate storage for a partition at some point and > then later c

Re: [HACKERS] On partitioning

2014-12-10 Thread Amit Langote
> From: Robert Haas [mailto:robertmh...@gmail.com] > On Mon, Dec 8, 2014 at 2:56 PM, Andres Freund > wrote: > >> I don't think that's mutually exclusive with the idea of > >> partitions-as-tables. I mean, you can add code to the ALTER TABLE > >> path that says if (i_am_not_the_partitioning_root

Re: [HACKERS] On partitioning

2014-12-10 Thread Robert Haas
On Mon, Dec 8, 2014 at 10:59 PM, Amit Kapila wrote: > Yeah and also how would user specify the values, as an example > assume that table is partitioned on monthly_salary, so partition > definition would look: > > PARTITION BY LIST(monthly_salary) > ( > PARTITION salary_less_than_thousand VALUES(30

Re: [HACKERS] On partitioning

2014-12-10 Thread Robert Haas
On Mon, Dec 8, 2014 at 5:05 PM, Jim Nasby wrote: > Agreed, but it's possible to keep a block/CTID interface while doing > something different on the disk. Objection: hand-waving. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hack

Re: [HACKERS] On partitioning

2014-12-10 Thread Robert Haas
On Wed, Dec 10, 2014 at 9:22 AM, Alvaro Herrera wrote: > The problem with naming partitions is that the user has to pick names > for every partition, which is tedious and doesn't provide any > significant benefit. The input I had from users of other partitioning > systems was that they very much

Re: [HACKERS] On partitioning

2014-12-10 Thread Alvaro Herrera
Amit Langote wrote: > On Wed, Dec 10, 2014 at 12:46 PM, Amit Kapila wrote: > > On Tue, Dec 9, 2014 at 7:21 PM, Alvaro Herrera > > wrote: > >> FWIW in my original proposal I was rejecting some things that after > >> further consideration turn out to be possible to allow; for instance > >> direct

Re: [HACKERS] On partitioning

2014-12-09 Thread Amit Langote
On Wed, Dec 10, 2014 at 12:46 PM, Amit Kapila wrote: > On Tue, Dec 9, 2014 at 7:21 PM, Alvaro Herrera > wrote: >> >> Amit Kapila wrote: >> > On Tue, Dec 9, 2014 at 1:42 AM, Robert Haas >> > wrote: >> > > On Mon, Dec 8, 2014 at 2:56 PM, Andres Freund >> > wrote: >> > > >> I don't think that's

Re: [HACKERS] On partitioning

2014-12-09 Thread Amit Langote
On Wed, Dec 10, 2014 at 12:33 PM, Amit Kapila wrote: > On Tue, Dec 9, 2014 at 11:44 PM, Josh Berkus wrote: >> On 12/09/2014 12:17 AM, Amit Langote wrote: >> >> Now if user wants to define multi-column Partition based on >> >> > monthly_salary and annual_salary, how do we want him to >> >> > spe

Re: [HACKERS] On partitioning

2014-12-09 Thread Amit Kapila
On Tue, Dec 9, 2014 at 7:21 PM, Alvaro Herrera wrote: > > Amit Kapila wrote: > > On Tue, Dec 9, 2014 at 1:42 AM, Robert Haas wrote: > > > On Mon, Dec 8, 2014 at 2:56 PM, Andres Freund > > wrote: > > > >> I don't think that's mutually exclusive with the idea of > > > >> partitions-as-tables. I m

Re: [HACKERS] On partitioning

2014-12-09 Thread Amit Kapila
On Tue, Dec 9, 2014 at 11:44 PM, Josh Berkus wrote: > On 12/09/2014 12:17 AM, Amit Langote wrote: > >> Now if user wants to define multi-column Partition based on > >> > monthly_salary and annual_salary, how do we want him to > >> > specify the values. Basically how to distinguish which values >

Re: [HACKERS] On partitioning

2014-12-09 Thread Jim Nasby
On 12/8/14, 5:19 PM, Josh Berkus wrote: On 12/08/2014 02:12 PM, Jim Nasby wrote: On 12/8/14, 12:26 PM, Josh Berkus wrote: 4. Creation Locking Problem high probability of lock pile-ups whenever a new partition is created on demand due to multiple backends trying to create the partition at the sa

Re: [HACKERS] On partitioning

2014-12-09 Thread Josh Berkus
On 12/09/2014 12:17 AM, Amit Langote wrote: >> Now if user wants to define multi-column Partition based on >> > monthly_salary and annual_salary, how do we want him to >> > specify the values. Basically how to distinguish which values >> > belong to first column key and which one's belong to secon

Re: [HACKERS] On partitioning

2014-12-09 Thread Alvaro Herrera
Amit Kapila wrote: > On Tue, Dec 9, 2014 at 1:42 AM, Robert Haas wrote: > > On Mon, Dec 8, 2014 at 2:56 PM, Andres Freund > wrote: > > >> I don't think that's mutually exclusive with the idea of > > >> partitions-as-tables. I mean, you can add code to the ALTER TABLE > > >> path that says if (i_

Re: [HACKERS] On partitioning

2014-12-09 Thread Alvaro Herrera
Josh Berkus wrote: Hi, > Pardon me for jumping into this late. In general, I like Alvaro's > approach. Please don't call this "Alvaro's approach" as I'm not involved in this anymore. Amit Langote has taken ownership of it now. While some resemblance to what I originally proposed might remain,

Re: [HACKERS] On partitioning

2014-12-09 Thread Amit Langote
On Tue, Dec 9, 2014 at 12:59 PM, Amit Kapila wrote: > On Tue, Dec 9, 2014 at 8:08 AM, Amit Langote > wrote: >> > From: Robert Haas [mailto:robertmh...@gmail.com] >> > On Sat, Dec 6, 2014 at 2:59 AM, Amit Kapila >> > wrote: >> > >> I guess you could list or hash partition on multiple columns, t

Re: [HACKERS] On partitioning

2014-12-08 Thread Amit Langote
On Tue, Dec 9, 2014 at 12:59 PM, Amit Kapila wrote: > On Tue, Dec 9, 2014 at 8:08 AM, Amit Langote > wrote: >> > From: Robert Haas [mailto:robertmh...@gmail.com] >> > I don't understand. If you want to range partition on columns (a, b), >> > you say that, say, tuples with (a, b) values less tha

Re: [HACKERS] On partitioning

2014-12-08 Thread Amit Kapila
On Tue, Dec 9, 2014 at 1:42 AM, Robert Haas wrote: > On Mon, Dec 8, 2014 at 2:56 PM, Andres Freund wrote: > >> I don't think that's mutually exclusive with the idea of > >> partitions-as-tables. I mean, you can add code to the ALTER TABLE > >> path that says if (i_am_not_the_partitioning_root) e

Re: [HACKERS] On partitioning

2014-12-08 Thread Amit Kapila
On Tue, Dec 9, 2014 at 8:08 AM, Amit Langote wrote: > > From: Robert Haas [mailto:robertmh...@gmail.com] > > On Sat, Dec 6, 2014 at 2:59 AM, Amit Kapila > > wrote: > > >> I guess you could list or hash partition on multiple columns, too. > > > > > > How would you distinguish values in list partit

Re: [HACKERS] On partitioning

2014-12-08 Thread Amit Langote
> From: Robert Haas [mailto:robertmh...@gmail.com] > On Sat, Dec 6, 2014 at 2:59 AM, Amit Kapila > wrote: > >> I guess you could list or hash partition on multiple columns, too. > > > > How would you distinguish values in list partition for multiple > > columns? I mean for range partition, we are

Re: [HACKERS] On partitioning

2014-12-08 Thread Josh Berkus
On 12/08/2014 02:12 PM, Jim Nasby wrote: > On 12/8/14, 12:26 PM, Josh Berkus wrote: >> 4. Creation Locking Problem >> high probability of lock pile-ups whenever a new partition is created on >> demand due to multiple backends trying to create the partition at the >> same time. >> Not Addressed? >

Re: [HACKERS] On partitioning

2014-12-08 Thread Jim Nasby
On 12/8/14, 12:26 PM, Josh Berkus wrote: 4. Creation Locking Problem high probability of lock pile-ups whenever a new partition is created on demand due to multiple backends trying to create the partition at the same time. Not Addressed? Do users actually try and create new partitions during DM

Re: [HACKERS] On partitioning

2014-12-08 Thread Jim Nasby
On 12/8/14, 1:05 PM, Robert Haas wrote: Besides, I haven't really seen anyone propose something that sounds like a credible alternative. If we could make partition objects things that the storage layer needs to know about but the query planner doesn't need to understand, that'd be maybe worth co

Re: [HACKERS] On partitioning

2014-12-08 Thread Robert Haas
On Mon, Dec 8, 2014 at 2:58 PM, Josh Berkus wrote: >> I think any new partitioning system should keep the good things about >> the existing system, of which there are some, and not try to reinvent >> the wheel. The yard stick for a new system shouldn't be "is this >> different enough?" but "does

Re: [HACKERS] On partitioning

2014-12-08 Thread Robert Haas
On Mon, Dec 8, 2014 at 2:56 PM, Andres Freund wrote: >> I don't think that's mutually exclusive with the idea of >> partitions-as-tables. I mean, you can add code to the ALTER TABLE >> path that says if (i_am_not_the_partitioning_root) ereport(ERROR, ...) >> wherever you want. > > That'll be a lo

Re: [HACKERS] On partitioning

2014-12-08 Thread Josh Berkus
On 12/08/2014 11:40 AM, Robert Haas wrote: >> I don't thing its feasible to drop inheritance partitioning at this >> point; too many user exploit a lot of peculiarities of that system which >> wouldn't be supported by any other system. So any new partitioning >> system we're talking about would be

Re: [HACKERS] On partitioning

2014-12-08 Thread Andres Freund
On 2014-12-08 14:48:50 -0500, Robert Haas wrote: > On Mon, Dec 8, 2014 at 2:39 PM, Andres Freund wrote: > >> I guess I'm in disagreement with you - and, perhaps - the majority on > >> this point. I think that ship has already sailed: partitions ARE > >> tables. We can try to make it less necessa

Re: [HACKERS] On partitioning

2014-12-08 Thread Robert Haas
On Mon, Dec 8, 2014 at 2:39 PM, Andres Freund wrote: >> I guess I'm in disagreement with you - and, perhaps - the majority on >> this point. I think that ship has already sailed: partitions ARE >> tables. We can try to make it less necessary for users to ever look >> at those tables as separate

Re: [HACKERS] On partitioning

2014-12-08 Thread Robert Haas
On Mon, Dec 8, 2014 at 2:30 PM, Josh Berkus wrote: > On 12/08/2014 11:05 AM, Robert Haas wrote: >> I guess I'm in disagreement with you - and, perhaps - the majority on >> this point. I think that ship has already sailed: partitions ARE >> tables. We can try to make it less necessary for users t

Re: [HACKERS] On partitioning

2014-12-08 Thread Andres Freund
On 2014-12-08 14:05:52 -0500, Robert Haas wrote: > On Sat, Dec 6, 2014 at 3:06 AM, Amit Kapila wrote: > > Sure, I don't feel we should not provide anyway to take dump > > for individual partition but not at level of independent table. > > May be something like --table > > --partition . > > > > In

Re: [HACKERS] On partitioning

2014-12-08 Thread Josh Berkus
On 12/08/2014 11:05 AM, Robert Haas wrote: > I guess I'm in disagreement with you - and, perhaps - the majority on > this point. I think that ship has already sailed: partitions ARE > tables. We can try to make it less necessary for users to ever look > at those tables as separate objects, and I

Re: [HACKERS] On partitioning

2014-12-08 Thread Robert Haas
On Sat, Dec 6, 2014 at 3:06 AM, Amit Kapila wrote: > Sure, I don't feel we should not provide anyway to take dump > for individual partition but not at level of independent table. > May be something like --table > --partition . > > In general, I think we should try to avoid exposing that partitio

Re: [HACKERS] On partitioning

2014-12-08 Thread Robert Haas
On Mon, Dec 8, 2014 at 12:13 AM, Amit Langote wrote: > So just to clarify, first and last destinations are considered "defined" if > you have something like: > > ... > PARTITION p1 VALUES LESS THAN 10 > PARTITION p2 VALUES BETWEEN 10 AND 20 > PARTITION p3 VALUES GREATER THAN 20 > ... > > And "not

Re: [HACKERS] On partitioning

2014-12-08 Thread Robert Haas
On Sat, Dec 6, 2014 at 2:59 AM, Amit Kapila wrote: >> I guess you could list or hash partition on multiple columns, too. > > How would you distinguish values in list partition for multiple > columns? I mean for range partition, we are sure there will > be either one value for each column, but for

Re: [HACKERS] On partitioning

2014-12-08 Thread Josh Berkus
All, Pardon me for jumping into this late. In general, I like Alvaro's approach. However, I wanted to list the major shortcomings of the existing replication system (based on complaints by PGX's users and on IRC) and compare them to Alvaro's proposed implementation to make sure that enough of th

Re: [HACKERS] On partitioning

2014-12-07 Thread Amit Langote
From: Amit Kapila [mailto:amit.kapil...@gmail.com] > > > How would you distinguish values in list partition for multiple > > > columns? I mean for range partition, we are sure there will > > > be either one value for each column, but for list it could > > > be multiple and not fixed for each part

Re: [HACKERS] On partitioning

2014-12-07 Thread Amit Kapila
On Mon, Dec 8, 2014 at 11:01 AM, Amit Langote wrote: > From: Amit Kapila [mailto:amit.kapil...@gmail.com] > Sent: Saturday, December 06, 2014 5:00 PM > To: Robert Haas > Cc: Amit Langote; Andres Freund; Alvaro Herrera; Bruce Momjian; Pg Hackers > Subject: Re: [HACKERS] On partitio

Re: [HACKERS] On partitioning

2014-12-07 Thread Amit Langote
From: pgsql-hackers-ow...@postgresql.org [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Amit Kapila Sent: Saturday, December 06, 2014 5:06 PM To: Robert Haas Cc: Amit Langote; Andres Freund; Alvaro Herrera; Bruce Momjian; Pg Hackers Subject: Re: [HACKERS] On partitioning On Fri, Dec

Re: [HACKERS] On partitioning

2014-12-07 Thread Amit Langote
From: Amit Kapila [mailto:amit.kapil...@gmail.com] Sent: Saturday, December 06, 2014 5:00 PM To: Robert Haas Cc: Amit Langote; Andres Freund; Alvaro Herrera; Bruce Momjian; Pg Hackers Subject: Re: [HACKERS] On partitioning On Fri, Dec 5, 2014 at 10:03 PM, Robert Haas wrote: > On Tue, De

Re: [HACKERS] On partitioning

2014-12-07 Thread Amit Langote
Hi Robert, > From: Robert Haas [mailto:robertmh...@gmail.com] > On Tue, Dec 2, 2014 at 10:43 PM, Amit Langote > wrote: > >> So, we're going to support exactly two levels of partitioning? > >> partitions with partissub=false and subpartitions with partissub=true? > >> Why not support only one lev

Re: [HACKERS] On partitioning

2014-12-06 Thread Amit Kapila
On Fri, Dec 5, 2014 at 10:12 PM, Robert Haas wrote: > On Fri, Dec 5, 2014 at 2:18 AM, Amit Kapila wrote: > > Do we really need to support dml or pg_dump for individual partitions? > > I think we do. It's quite reasonable for a DBA (or developer or > whatever) to want to dump all the data that's

Re: [HACKERS] On partitioning

2014-12-06 Thread Amit Kapila
On Fri, Dec 5, 2014 at 10:03 PM, Robert Haas wrote: > On Tue, Dec 2, 2014 at 10:43 PM, Amit Langote > wrote: > > > I wonder if your suggestion of pg_node_tree plays well here. This then could be a list of CONSTs or some such... And I am thinking it's a concern only for range partitions, no? (that

Re: [HACKERS] On partitioning

2014-12-05 Thread Robert Haas
On Fri, Dec 5, 2014 at 3:05 PM, Jim Nasby wrote: >> On what basis do you expect that? Every time you use a view, you're >> using a pg_node_tree. Nobody's ever complained that having to reload >> the pg_node_tree column was too slow, and I see no reason to suppose >> that things would be any diff

Re: [HACKERS] On partitioning

2014-12-05 Thread Jim Nasby
On 12/5/14, 2:02 PM, Robert Haas wrote: On Fri, Dec 5, 2014 at 2:52 PM, Jim Nasby wrote: The other option would be to use some custom rowtype to store boundary values and have a method that can form a boundary tuple from a real one. Either way, I suspect this is better than frequently evaluatin

Re: [HACKERS] On partitioning

2014-12-05 Thread Robert Haas
On Fri, Dec 5, 2014 at 2:52 PM, Jim Nasby wrote: > The other option would be to use some custom rowtype to store boundary > values and have a method that can form a boundary tuple from a real one. > Either way, I suspect this is better than frequently evaluating > pg_node_trees. On what basis do

Re: [HACKERS] On partitioning

2014-12-05 Thread Jim Nasby
On 12/5/14, 1:22 PM, Jim Nasby wrote: On 12/5/14, 3:42 AM, Amit Langote wrote: > I think you are right. I think in this case we need something similar >to column pg_index.indexprs which is of type pg_node_tree(which >seems to be already suggested by Robert). So may be we can proceed >with this

Re: [HACKERS] On partitioning

2014-12-05 Thread Jim Nasby
On 12/5/14, 3:42 AM, Amit Langote wrote: > I think you are right. I think in this case we need something similar >to column pg_index.indexprs which is of type pg_node_tree(which >seems to be already suggested by Robert). So may be we can proceed >with this type and see if any one else has bette

Re: [HACKERS] On partitioning

2014-12-05 Thread Robert Haas
On Fri, Dec 5, 2014 at 3:11 AM, Amit Langote wrote: >> I think you are right. I think in this case we need something similar >> to column pg_index.indexprs which is of type pg_node_tree(which >> seems to be already suggested by Robert). So may be we can proceed >> with this type and see if any on

  1   2   >