Re: [HACKERS] Dynamic Partitioning using Segment Visibility Maps

2008-01-04 Thread Simon Riggs
On Fri, 2008-01-04 at 13:29 +0100, Markus Schiltknecht wrote: Given that we are operating on segments here, to which the DBA has very limited information and access, I prefer the term Segment Exclusion. I think of that as an optimization of sequential scans on tables with the above

Re: [HACKERS] Dynamic Partitioning using Segment Visibility Maps

2008-01-04 Thread Markus Schiltknecht
Hello Simon, Simon Riggs wrote: I've come up with an alternative concept to allow us to discuss the particular merits of each. ISTM that this new proposal has considerable potential. Hm.. interesting idea. If we were able to keep track of which sections of a table are now read-only then we

Re: [HACKERS] Dynamic Partitioning using Segment Visibility Maps

2008-01-04 Thread Markus Schiltknecht
Hi, Simon Riggs wrote: - any Fact table where measurements/observations/events are accumulated e.g. Web Hits (any Internet events) Call Detail Records Sales Security Events Scientific Measurements Process Control - any Major Entity where new entities are created from a sequence e.g. Orders,

Re: [HACKERS] Dynamic Partitioning using Segment Visibility Maps

2008-01-04 Thread Markus Schiltknecht
Hi, Simon Riggs wrote: The smaller the partition size the greater the overhead of managing it. Also I've been looking at read-only tables and compression, as you may know. My idea was that in the future we could mark segments as either - read-only - compressed - able to be shipped off to

Re: [HACKERS] Dynamic Partitioning using Segment Visibility Maps

2008-01-04 Thread Andrew Sullivan
On Fri, Jan 04, 2008 at 01:29:55PM +0100, Markus Schiltknecht wrote: Agreed. Just a minor note: I find marked read-only too strong, as it implies an impossibility to write. I propose speaking about mostly-read segments, or optimized for reading or similar. I do want some segments to be

Re: [HACKERS] Dynamic Partitioning using Segment Visibility Maps

2008-01-04 Thread Simon Riggs
On Fri, 2008-01-04 at 13:06 -0500, Andrew Sullivan wrote: On Fri, Jan 04, 2008 at 01:29:55PM +0100, Markus Schiltknecht wrote: Agreed. Just a minor note: I find marked read-only too strong, as it implies an impossibility to write. I propose speaking about mostly-read segments, or

Re: [HACKERS] Dynamic Partitioning using Segment Visibility Maps

2008-01-04 Thread Andrew Sullivan
On Fri, Jan 04, 2008 at 10:26:54PM +0100, Markus Schiltknecht wrote: I'm still puzzled about how a DBA is expected to figure out which segments to mark. I think that part might be hand-wavy still. But once this facility is there, what's to prevent the current active segment (and the rest)

Re: [HACKERS] Dynamic Partitioning using Segment Visibility Maps

2008-01-04 Thread Markus Schiltknecht
Hi, Simon Riggs wrote: On Fri, 2008-01-04 at 13:06 -0500, Andrew Sullivan wrote: On Fri, Jan 04, 2008 at 01:29:55PM +0100, Markus Schiltknecht wrote: Agreed. Just a minor note: I find marked read-only too strong, as it implies an impossibility to write. I propose speaking about mostly-read

Re: [HACKERS] Dynamic Partitioning using Segment Visibility Maps

2008-01-04 Thread Simon Riggs
On Fri, 2008-01-04 at 22:26 +0100, Markus Schiltknecht wrote: I'm still puzzled about how a DBA is expected to figure out which segments to mark. Simon, are you assuming we are going to pass on segment numbers to the DBA one day? No Way! That would stop Richard's idea to make the segment

Re: [HACKERS] Dynamic Partitioning using Segment Visibility Maps

2008-01-04 Thread Robert Treat
On Friday 04 January 2008 17:01, Simon Riggs wrote: On Fri, 2008-01-04 at 22:26 +0100, Markus Schiltknecht wrote: I'm still puzzled about how a DBA is expected to figure out which segments to mark. Simon, are you assuming we are going to pass on segment numbers to the DBA one day? No Way!

Re: [HACKERS] Dynamic Partitioning using Segment Visibility Maps

2008-01-03 Thread Simon Riggs
On Thu, 2008-01-03 at 00:41 +, Sam Mason wrote: On Wed, Jan 02, 2008 at 05:56:14PM +, Simon Riggs wrote: Like it? Sounds good. I've only given it a quick scan though. Would read-only segments retain the same disk-level format as is currently? Yes, no changes at all to the

Re: [HACKERS] Dynamic Partitioning using Segment Visibility Maps

2008-01-03 Thread Gokulakannan Somasundaram
Hi Simon, Looks like a novel idea. I just want to confirm my understanding of the proposal. a) This proposal would work for the kind of table organizations which are currently partitioned and maintained based on some kind of timestamp. Consider one of the use-case. A large Retail firm

Re: [HACKERS] Dynamic Partitioning using Segment Visibility Maps

2008-01-02 Thread Sam Mason
On Wed, Jan 02, 2008 at 05:56:14PM +, Simon Riggs wrote: Like it? Sounds good. I've only given it a quick scan though. Would read-only segments retain the same disk-level format as is currently? It seems possible to remove the MVCC fields and hence get more tuples per page--- whether this

Re: [HACKERS] Postgresql partitioning problems suggested solutions

2007-10-13 Thread Gregory Stark
SHARMILA JOTHIRAJAH [EMAIL PROTECTED] writes: Now for SELECT * FROM Foo WHERE b='2' it should know to access only Foo_2, I suppose it could be done with a rule, but that should work even if b='2' is implicitly given (not just if b = constant) is stated explicitly. This much already exists

Re: [HACKERS] 2D partitioning of VLDB - sane or not?

2007-08-14 Thread Zeugswetter Andreas ADI SD
Which brings us back to the original issue. If I decide to stick with the current implementation and not improve our existing partitioning mechanisms to scale to 100,000 partitions, I could do something like: There is a point where you can leave the selection of the correct rows to normal

Re: [HACKERS] 2D partitioning of VLDB - sane or not?

2007-08-14 Thread Gregory Stark
Zeugswetter Andreas ADI SD [EMAIL PROTECTED] writes: I'd say that that point currently is well below 2000 partitions for all common db systems. I think it will depend heavily on the type of queries you're talking about. Postgres's constraint_exclusion is a linear search and does quite a bit of

Re: [HACKERS] 2D partitioning of VLDB - sane or not?

2007-08-13 Thread Jason Nerothin
Josh, I think what you are suggesting is something like this: -- begin SQL -- core=# CREATE TABLE temp_x( x_id BIGINT PRIMARY KEY, x_info VARCHAR(16) NOT NULL DEFAULT 'x_info'); NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index temp_x_pkey for table temp_x CREATE TABLE core=# CREATE

Re: [HACKERS] 2D partitioning of VLDB - sane or not?

2007-08-13 Thread Josh Berkus
Jason, Which brings us back to the original issue. If I decide to stick with the current implementation and not improve our existing partitioning mechanisms to scale to 100,000 partitions, I could do something like: Maintain 2 copies of the parent table (partitioned by 256). Inherit from

Re: [HACKERS] 2D partitioning of VLDB - sane or not?

2007-08-11 Thread Josh Berkus
Jason, Aside from running into a known bug with too many triggers when creating gratuitous indices on these tables, I feel as it may be possible to do what I want without breaking everything. But then again, am I taking too many liberties with technology that maybe didn't have use cases like

Re: [HACKERS] table partitioning pl/pgsql helpers

2007-06-01 Thread Jim Nasby
Dropping -hackers; that list is for development of the database engine itself. The problem is that rules will happen before triggers, so what you're trying to do will never work. Instead, just have the trigger insert the data into the appropriate table. On May 30, 2007, at 9:55 AM,

Re: [HACKERS] Auto Partitioning

2007-04-06 Thread Tom Lane
Markus Schiltknecht [EMAIL PROTECTED] writes: Uh.. can you elaborate on that? AFAICS, you would simply have to query multiple btree indexes and make sure non of them is violated. That only works for the partition-key indexes, ie, ones where you can be sure a-priori that there cannot be

Re: [HACKERS] Auto Partitioning

2007-04-06 Thread Simon Riggs
On Fri, 2007-04-06 at 01:56 -0400, Tom Lane wrote: Markus Schiltknecht [EMAIL PROTECTED] writes: Uh.. can you elaborate on that? AFAICS, you would simply have to query multiple btree indexes and make sure non of them is violated. That only works for the partition-key indexes, ie, ones

Re: [HACKERS] Auto Partitioning

2007-04-06 Thread Gregory Stark
Simon Riggs [EMAIL PROTECTED] writes: If we partition on invoice_date only, there is an implication that people will search for invoices on date range only too, otherwise why not just partition on invoice_id. This still works with the compound key approach. Well there are practical problems

Re: [HACKERS] Auto Partitioning

2007-04-06 Thread Zeugswetter Andreas ADI SD
(But that sounds rather like pie in the sky, actually. Which other databases can do that, and how do they do it?) Oracle does it, by building a big index. Few people use it. And others allow a different partitioning strategy for each index, but that has the same problem of how to remove

Re: [HACKERS] Auto Partitioning

2007-04-06 Thread Simon Riggs
On Fri, 2007-04-06 at 12:47 +0200, Zeugswetter Andreas ADI SD wrote: What I think we would like to have is putting the append nodes into an order that allows removing the sort node whenever that can be done. And maybe a merge node (that replaces the append and sort node) that can merge

Re: [HACKERS] Auto Partitioning

2007-04-06 Thread Markus Schiltknecht
Simon Riggs wrote: i.e. if we have partitions for each year (2001, 2002, 2003 2004, 2005, 2006, 2007) AND we have already proved that 2005 is excluded when we have a WHERE clause saying year = 2006, then we should be able to use the ordering to prove that partitions for 2004 and before are also

Re: [HACKERS] Auto Partitioning

2007-04-06 Thread Joshua D. Drake
Zeugswetter Andreas ADI SD wrote: (But that sounds rather like pie in the sky, actually. Which other databases can do that, and how do they do it?) Oracle does it, by building a big index. Few people use it. And others allow a different partitioning strategy for each index, but that has the

Re: [HACKERS] Auto Partitioning

2007-04-06 Thread Joshua D. Drake
But if we could find a way to represent that it would make a lot of common use cases much more convenient to use. (But that sounds rather like pie in the sky, actually. Which other databases can do that, and how do they do it?) Oracle does it, by building a big index. Few people use it.

Re: [HACKERS] Auto Partitioning

2007-04-06 Thread David Fetter
On Fri, Apr 06, 2007 at 09:22:55AM -0700, Joshua D. Drake wrote: The people that use it are the people stuck by dogmatic rules about every table must have a primary key or every logical constraint must be protected by a database constraint. Ie, database shops run by the CYA principle. Or

Re: [HACKERS] Auto Partitioning

2007-04-06 Thread Joshua D. Drake
David Fetter wrote: On Fri, Apr 06, 2007 at 09:22:55AM -0700, Joshua D. Drake wrote: The people that use it are the people stuck by dogmatic rules about every table must have a primary key or every logical constraint must be protected by a database constraint. Ie, database shops run by the CYA

Re: [HACKERS] Auto Partitioning

2007-04-06 Thread Richard Troy
David Fetter wrote: On Fri, Apr 06, 2007 at 09:22:55AM -0700, Joshua D. Drake wrote: The people that use it are the people stuck by dogmatic rules about every table must have a primary key or every logical constraint must be protected by a database constraint. Ie, database shops run

Re: [HACKERS] Auto Partitioning

2007-04-06 Thread Simon Riggs
On Fri, 2007-04-06 at 16:08 +0200, Markus Schiltknecht wrote: Simon Riggs wrote: i.e. if we have partitions for each year (2001, 2002, 2003 2004, 2005, 2006, 2007) AND we have already proved that 2005 is excluded when we have a WHERE clause saying year = 2006, then we should be able to use

Re: [HACKERS] Auto Partitioning

2007-04-05 Thread Martijn van Oosterhout
On Wed, Apr 04, 2007 at 09:34:03PM +0200, Markus Schiltknecht wrote: Joshua D. Drake wrote: If we don't have multi-table indexes how do we enforce a primary key against a partitioned set? The executor would have to be clever enough to not do a single index scan, but possibly scan through

Re: [HACKERS] Auto Partitioning

2007-04-05 Thread NikhilS
Hi, The only problem I have with this is that the shops I know with big partitioned tables favor triggers over rules for both performance reason and a cleaner implementation. Even with automated rule creation this isnt going to change afaics... not to mention we already create our rules

Re: [HACKERS] Auto Partitioning

2007-04-05 Thread NikhilS
Hi, I had raised this issue about rules/triggers back then and the responses seemed to be evenly split as to which ones to use. Presumably your implementation already uses Triggers for INSERTs though, so why not use triggers for everything? No I am using rules for all the 3 cases. I am

Re: [HACKERS] Auto Partitioning

2007-04-05 Thread Simon Riggs
On Thu, 2007-04-05 at 13:59 +0530, NikhilS wrote: Hi, The only problem I have with this is that the shops I know with big partitioned tables favor triggers over rules for both performance reason and a cleaner implementation. Even with automated

Re: [HACKERS] Auto Partitioning

2007-04-05 Thread Simon Riggs
On Thu, 2007-04-05 at 16:35 +0530, NikhilS wrote: Hi, I had raised this issue about rules/triggers back then and the responses seemed to be evenly split as to which ones to use. Presumably your implementation already uses

Re: [HACKERS] Auto Partitioning

2007-04-05 Thread NikhilS
So we are unable to load any of the tables using COPY. Aww, guess should have stuck to triggers as a first choice. Mea culpa, since I should have investigated some more before deciding on rules, or should have prodded you more earlier:) Regards, Nikhils -- EnterpriseDB

Re: [HACKERS] Auto Partitioning

2007-04-05 Thread Robert Treat
On Wednesday 04 April 2007 21:17, Gregory Stark wrote: Simon Riggs [EMAIL PROTECTED] writes: If we don't have multi-table indexes how do we enforce a primary key against a partitioned set? What about non primary keys that are just UNIQUE? What about check constraints that aren't apart of

Re: [HACKERS] Auto Partitioning

2007-04-05 Thread Zeugswetter Andreas ADI SD
That lets you enforce unique constraints as long as the partition key is part of the unique constraint. Is that already sufficient? yes That would alter the ordering of the columns in the index, no? I mean: It produces ordered blocks of append nodes for range queries that span

Re: [HACKERS] Auto Partitioning

2007-04-05 Thread Markus Schiltknecht
Hi, Zeugswetter Andreas ADI SD wrote: CREATE INDEX x ON test(a, b, c); isn't the same as CRETAE INDEX x ON test(c, b, a); That is only a problem if you also want to avoid a sort (e.g. for an order by), ..or if you want to use that index for 'WHERE a = 5'. The first one is probably

Re: [HACKERS] Auto Partitioning

2007-04-05 Thread Markus Schiltknecht
Hi, Martijn van Oosterhout wrote: The executor would have to be clever enough to not do a single index scan, but possibly scan through multiple indexes when asking for uniqueness, depending on the partitioning rule set. But it's not the executor that checks uniqueness, it's built into the

Re: [HACKERS] Auto Partitioning

2007-04-05 Thread Martijn van Oosterhout
On Thu, Apr 05, 2007 at 10:00:37PM +0200, Markus Schiltknecht wrote: If someone manages to crack uniqueness for GiST indexes, we'll have our answer, since it has exactly the same problem but on a different scale. (Or vice-versa, if some gets uniqueness for multiple indexes, we can do it for

Re: [HACKERS] Auto Partitioning

2007-04-04 Thread Markus Schiltknecht
Hi, NikhilS wrote: The following things are TODOs: iv) Auto generate rules using the checks mentioned for the partitions, to handle INSERTs/DELETEs/UPDATEs to navigate them to the appropriate child. Note that checks specified directly on the master table will get inherited automatically.

Re: [HACKERS] Auto Partitioning

2007-04-04 Thread NikhilS
Hi, I appreciate you efforts, but I'm not sure if this has been discussed Thanks Markus. enough. There seem to be two ideas floating around: - you are heading for automating the current kludge, which involves creating partitions and constraints by hand. AFAICT, you want to

Re: [HACKERS] Auto Partitioning

2007-04-04 Thread Gregory Stark
Simon Riggs [EMAIL PROTECTED] writes: On Wed, 2007-04-04 at 14:20 +0200, Markus Schiltknecht wrote: Both proposals do not have much to do with the missing multi-table indices. It's clear to me that we have to implement those someday, anyway. I agree with much of your post, though this

Re: [HACKERS] Auto Partitioning

2007-04-04 Thread Andrew Dunstan
Simon Riggs wrote: My hope is to have a mechanism to partition indexes or recognise that they are partitioned, so that a set of provably-distinct unique indexes can provide the exact same functionlity as a single large unique index, just without the management nightmare. Will this address

Re: [HACKERS] Auto Partitioning

2007-04-04 Thread Markus Schiltknecht
Hi, NikhilS wrote: Our current partitioning solution is based on inheritance. With that in mind, for 8.3 I thought an implementation based on auto rules creation would be the way to go. That's completely reasonable. And as I've said, it's probably even a step towards what I've outlined

Re: [HACKERS] Auto Partitioning

2007-04-04 Thread Markus Schiltknecht
Hi, Simon Riggs wrote: I agree with much of your post, though this particular point caught my eye. If you'll forgive me for jumping on an isolated point in your post: No problem. Multi-table indexes sound like a good solution until you consider how big they would be. The reason we need a

Re: [HACKERS] Auto Partitioning

2007-04-04 Thread Markus Schiltknecht
Hi, Gregory Stark wrote: Put another way, multi-table indexes defeat the whole purpose of having partitioned the table in the first place. If you could have managed a single massive index then you wouldn't have bothered partitioning. That depends very much on the implementation of the

Re: [HACKERS] Auto Partitioning

2007-04-04 Thread Simon Riggs
On Wed, 2007-04-04 at 16:31 +0200, Markus Schiltknecht wrote: But how to create an (unique) index which is completely different from the partitioning key? Don't? Most high volume tables are Fact tables with potentially more than 1 row per Object/Dimension, so the unique index isn't

Re: [HACKERS] Auto Partitioning

2007-04-04 Thread Markus Schiltknecht
Hi, Simon Riggs wrote: Most high volume tables are Fact tables with potentially more than 1 row per Object/Dimension, so the unique index isn't appropriate in those cases. When partitioning a Major Entity its much easier to regard the PK as the partitioning key + unique key, which is

Re: [HACKERS] Auto Partitioning

2007-04-04 Thread David Fetter
On Wed, Apr 04, 2007 at 10:07:39AM -0400, Andrew Dunstan wrote: Simon Riggs wrote: My hope is to have a mechanism to partition indexes or recognise that they are partitioned, so that a set of provably-distinct unique indexes can provide the exact same functionlity as a single large unique

Re: [HACKERS] Auto Partitioning

2007-04-04 Thread Gregory Stark
Markus Schiltknecht [EMAIL PROTECTED] writes: CREATE INDEX x ON test(a, b, c); That's why I'd say, the first column of an index would have to be equal to all of the columns used in the partitioning key. That's certainly the simple case. It would be nice to be able to create an index like

Re: [HACKERS] Auto Partitioning

2007-04-04 Thread Markus Schiltknecht
Hi, Gregory Stark wrote: However there are also cases such as where you have a=0..99 in one partition and a=100..199 in partition two, etc. It could still automatically build indexes on (a,b,c) on each partition and somehow note that the unique constraint is guaranteed across the whole

Re: [HACKERS] Auto Partitioning

2007-04-04 Thread Simon Riggs
On Wed, 2007-04-04 at 20:55 +0200, Markus Schiltknecht wrote: Questioning the other way around: do we need any sort of multi-table indexes at all, or isn't it enough to teach the planner and executor how to intelligently scan through (possibly) multiple indexes to get what is requested?

Re: [HACKERS] Auto Partitioning

2007-04-04 Thread Joshua D. Drake
Simon Riggs wrote: On Wed, 2007-04-04 at 20:55 +0200, Markus Schiltknecht wrote: Questioning the other way around: do we need any sort of multi-table indexes at all, or isn't it enough to teach the planner and executor how to intelligently scan through (possibly) multiple indexes to get what

Re: [HACKERS] Auto Partitioning

2007-04-04 Thread Andrew Dunstan
David Fetter wrote: On Wed, Apr 04, 2007 at 10:07:39AM -0400, Andrew Dunstan wrote: Simon Riggs wrote: My hope is to have a mechanism to partition indexes or recognise that they are partitioned, so that a set of provably-distinct unique indexes can provide the exact same functionlity

Re: [HACKERS] Auto Partitioning

2007-04-04 Thread Markus Schiltknecht
Hi, Joshua D. Drake wrote: If we don't have multi-table indexes how do we enforce a primary key against a partitioned set? The executor would have to be clever enough to not do a single index scan, but possibly scan through multiple indexes when asking for uniqueness, depending on the

Re: [HACKERS] Auto Partitioning

2007-04-04 Thread Markus Schiltknecht
Simon Riggs wrote: The planner already uses the Append node to put together multiple plans. The great thing is it will put together IndexScans and SeqScans as applicable. No need for multi-scans as a special node type. Yes... only that mixing 'concurrent' index scans in the right order would

Re: [HACKERS] Auto Partitioning

2007-04-04 Thread Markus Schiltknecht
Andrew Dunstan wrote: David Fetter wrote: That would be where the provably-distinct part comes in, so yes. That assumes you can provide some provably distinct test. In the general case I have in mind that isn't so. Could you please give a somewhat more concrete example, I'm not following

Re: [HACKERS] Auto Partitioning

2007-04-04 Thread Andrew Dunstan
Markus Schiltknecht wrote: Andrew Dunstan wrote: David Fetter wrote: That would be where the provably-distinct part comes in, so yes. That assumes you can provide some provably distinct test. In the general case I have in mind that isn't so. Could you please give a somewhat more concrete

Re: [HACKERS] Auto Partitioning

2007-04-04 Thread Markus Schiltknecht
Hi, Andrew Dunstan wrote: I guess my point was really that multi-table indexes might have uses beyond partitioning. Aha, now I understand. Thanks for the clarification. Say I have two tables, each with a field FKed to a field in a third table. I'd like to create the values to be unique

Re: [HACKERS] Auto Partitioning

2007-04-04 Thread Simon Riggs
On Wed, 2007-04-04 at 12:10 -0700, Joshua D. Drake wrote: Simon Riggs wrote: On Wed, 2007-04-04 at 20:55 +0200, Markus Schiltknecht wrote: Questioning the other way around: do we need any sort of multi-table indexes at all, or isn't it enough to teach the planner and executor how to

Re: [HACKERS] Auto Partitioning

2007-04-04 Thread Robert Treat
On Wednesday 04 April 2007 09:19, NikhilS wrote: Our current partitioning solution is based on inheritance. With that in mind, for 8.3 I thought an implementation based on auto rules creation would be the way to go. The only problem I have with this is that the shops I know with big

Re: [HACKERS] Auto Partitioning

2007-04-04 Thread Joshua D. Drake
Robert Treat wrote: On Wednesday 04 April 2007 09:19, NikhilS wrote: Our current partitioning solution is based on inheritance. With that in mind, for 8.3 I thought an implementation based on auto rules creation would be the way to go. The only problem I have with this is that the shops I

Re: [HACKERS] Auto Partitioning

2007-04-04 Thread Gregory Stark
Simon Riggs [EMAIL PROTECTED] writes: If we don't have multi-table indexes how do we enforce a primary key against a partitioned set? What about non primary keys that are just UNIQUE? What about check constraints that aren't apart of the exclusion? I can come up with arbitrary examples

Re: [HACKERS] Auto Partitioning Patch - WIP version 1

2007-04-03 Thread NikhilS
Hi, The following things are TODOs: iv) Auto generate rules using the checks mentioned for the partitions, to handle INSERTs/DELETEs/UPDATEs to navigate them to the appropriate child. Note that checks specified directly on the master table will get inherited automatically. Am planning to

Re: [HACKERS] Vertical Partitioning with TOAST

2006-03-04 Thread Hannu Krosing
Ühel kenal päeval, N, 2006-03-02 kell 22:15, kirjutas Bruce Momjian: Is there still interst in this idea for TODO? Just to voice my support - Yes, I think that being able to set lower thresolds for TOAST is very useful in several cases. Also getting rid of toast index and start using ctids

Re: [HACKERS] Vertical Partitioning with TOAST

2006-03-04 Thread Martijn van Oosterhout
On Sat, Mar 04, 2006 at 12:15:46PM +0200, Hannu Krosing wrote: Also getting rid of toast index and start using ctids directly would be a big bonus. When using direct ctids we could use either ctid chains or some sort of skiplist for access to N-th TOAST chunk. I suppose this would mean that

Re: [HACKERS] Vertical Partitioning with TOAST

2006-03-04 Thread Tom Lane
Martijn van Oosterhout kleptog@svana.org writes: On Sat, Mar 04, 2006 at 12:15:46PM +0200, Hannu Krosing wrote: Also getting rid of toast index and start using ctids directly would be a big bonus. When using direct ctids we could use either ctid chains or some sort of skiplist for access to

Re: [HACKERS] Vertical Partitioning with TOAST

2006-03-04 Thread Hannu Krosing
Ühel kenal päeval, L, 2006-03-04 kell 10:31, kirjutas Tom Lane: Martijn van Oosterhout kleptog@svana.org writes: On Sat, Mar 04, 2006 at 12:15:46PM +0200, Hannu Krosing wrote: Also getting rid of toast index and start using ctids directly would be a big bonus. When using direct ctids we

Re: [HACKERS] Vertical Partitioning with TOAST

2006-03-03 Thread Jim C. Nasby
If this would be accepted I might actually be able to accomplish this. Maybe. :) But having a TODO wouldn't be a bad idea as well... Would this require 2 new fields in pg_attribute, or is there a better way to store the thresholds? I'm thinking that each field would need two special values; 0 for

Re: [HACKERS] Vertical Partitioning with TOAST

2006-03-02 Thread Bruce Momjian
Is there still interst in this idea for TODO? --- Jim C. Nasby wrote: On Thu, Dec 08, 2005 at 10:03:43AM -0500, Bruce Momjian wrote: Jim C. Nasby wrote: On Thu, Dec 08, 2005 at 12:59:47AM -0500, Tom Lane wrote:

Re: [HACKERS] Vertical Partitioning with TOAST

2005-12-08 Thread Junji TERAMOTO
Hello all, Thank you for having the interest. Jim C. Nasby wrote: Valid point. I do think there's a lot of benefit to being able to set the limit much lower than what it currently defaults to today. We have a client that has a queue-type table that is updated very frequently. One of the

Re: [HACKERS] Vertical Partitioning with TOAST

2005-12-08 Thread Bruce Momjian
Jim C. Nasby wrote: On Thu, Dec 08, 2005 at 12:59:47AM -0500, Tom Lane wrote: Jim C. Nasby [EMAIL PROTECTED] writes: This seems like a useful feature to add, allowing for easy built-in verticle partitioning. Are there issues with the patch as-is? Other than the ones mentioned by the

Re: [HACKERS] Vertical Partitioning with TOAST

2005-12-08 Thread Jan Wieck
On 12/8/2005 1:42 PM, Jim C. Nasby wrote: On Thu, Dec 08, 2005 at 10:03:43AM -0500, Bruce Momjian wrote: Jim C. Nasby wrote: On Thu, Dec 08, 2005 at 12:59:47AM -0500, Tom Lane wrote: Jim C. Nasby [EMAIL PROTECTED] writes: This seems like a useful feature to add, allowing for easy

Re: [HACKERS] Vertical Partitioning with TOAST

2005-12-07 Thread Jim C. Nasby
This seems like a useful feature to add, allowing for easy built-in verticle partitioning. Are there issues with the patch as-is? (Other than it probably should have gone to -patches...) On Thu, Dec 01, 2005 at 05:59:08PM +0900, Junji TERAMOTO wrote: Hi all, I wrote a experimental patch for a

Re: [HACKERS] Vertical Partitioning with TOAST

2005-12-07 Thread Tom Lane
Jim C. Nasby [EMAIL PROTECTED] writes: This seems like a useful feature to add, allowing for easy built-in verticle partitioning. Are there issues with the patch as-is? Other than the ones mentioned by the poster? It seemed to me more like a not-too-successful experiment than something ready

Re: [HACKERS] Vertical Partitioning with TOAST

2005-12-07 Thread Jim C. Nasby
On Thu, Dec 08, 2005 at 12:59:47AM -0500, Tom Lane wrote: Jim C. Nasby [EMAIL PROTECTED] writes: This seems like a useful feature to add, allowing for easy built-in verticle partitioning. Are there issues with the patch as-is? Other than the ones mentioned by the poster? It seemed to me

Re: [HACKERS] Table Partitioning is in 8.1

2005-09-23 Thread Simon Riggs
On Thu, 2005-09-22 at 14:37 -0500, Jim C. Nasby wrote: On Thu, Sep 22, 2005 at 10:11:50AM +0100, Simon Riggs wrote: On Wed, 2005-09-21 at 15:39 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: Is it possible that the Release Notes do not fully explain the Constraint

Re: [HACKERS] Table Partitioning is in 8.1

2005-09-22 Thread Simon Riggs
On Wed, 2005-09-21 at 15:39 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: Is it possible that the Release Notes do not fully explain the Constraint Exclusion feature? Or is it the consensus that it works but not quite well enough to make a song and dance about yet? I

Re: [HACKERS] Table Partitioning is in 8.1

2005-09-22 Thread Jim C. Nasby
On Thu, Sep 22, 2005 at 10:11:50AM +0100, Simon Riggs wrote: On Wed, 2005-09-21 at 15:39 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: Is it possible that the Release Notes do not fully explain the Constraint Exclusion feature? Or is it the consensus that it works but

Re: [HACKERS] Table Partitioning is in 8.1

2005-09-22 Thread Joshua D. Drake
On reflection, the only changes I suggest are: 1) the phrase This allows for a type of table partitioning have the word basic inserted within it to become: This allows for a basic type of table partitioning How about just: Initial support for table partitioning. Yes it is non-committal but

Re: [HACKERS] Table Partitioning is in 8.1

2005-09-22 Thread Bruce Momjian
Word basic added. --- Joshua D. Drake wrote: On reflection, the only changes I suggest are: 1) the phrase This allows for a type of table partitioning have the word basic inserted within it to become: This

Re: [HACKERS] Table Partitioning is in 8.1

2005-09-22 Thread Bruce Momjian
Simon Riggs wrote: On Wed, 2005-09-21 at 15:39 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: Is it possible that the Release Notes do not fully explain the Constraint Exclusion feature? Or is it the consensus that it works but not quite well enough to make a song and

Re: [HACKERS] Table Partitioning is in 8.1

2005-09-21 Thread Martijn van Oosterhout
On Wed, Sep 21, 2005 at 06:10:15PM +0100, Simon Riggs wrote: Table Partitioning is in 8.1 I've just read Peter Eisentraut's presentation to the Dutch gov (very good BTW). On the last page I read that Table Partitioning is a future for PostgreSQLwhich is strange because Constraint

Re: [HACKERS] Table Partitioning is in 8.1

2005-09-21 Thread Tom Lane
Simon Riggs [EMAIL PROTECTED] writes: Is it possible that the Release Notes do not fully explain the Constraint Exclusion feature? Or is it the consensus that it works but not quite well enough to make a song and dance about yet? I hardly think that the existing constraint-exclusion code is

Re: [HACKERS] Table Partitioning is in 8.1

2005-09-21 Thread Hannu Krosing
On K, 2005-09-21 at 18:10 +0100, Simon Riggs wrote: Table Partitioning is in 8.1 I've just read Peter Eisentraut's presentation to the Dutch gov (very good BTW). On the last page I read that Table Partitioning is a future for PostgreSQLwhich is strange because Constraint Exclusion is an

Re: [HACKERS] Table Partitioning is in 8.1

2005-09-21 Thread Hannu Krosing
On K, 2005-09-21 at 15:39 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: Is it possible that the Release Notes do not fully explain the Constraint Exclusion feature? Or is it the consensus that it works but not quite well enough to make a song and dance about yet? I hardly

Re: [HACKERS] Table Partitioning, Part 1

2005-05-11 Thread Hannu Krosing
On T, 2005-05-10 at 23:09 +0100, Simon Riggs wrote: On Tue, 2005-05-10 at 16:31 +0300, Hannu Krosing wrote: If all partitions in the query had identical indexes on them, then we have another option. In that case, each index could be thought to form part of a larger index ordered

Re: [HACKERS] Table Partitioning, Part 1

2005-05-10 Thread Greg Stark
Tom Lane [EMAIL PROTECTED] writes: Partition Elimination relies upon being able to prove at execution time You mean plan time. Fwiw, both are possible. In oracle there are (at least) three different cases: 1. For queries like select * from tab the plan shows a multiple partition scan. 2.

Re: [HACKERS] Table Partitioning, Part 1

2005-05-10 Thread Hannu Krosing
On E, 2005-05-09 at 23:30 +0100, Simon Riggs wrote: A more in-depth consideration of the major design options and trade-offs available to us... this is an internals related discussion. Comments? 1. Embellish inheritance or separate infrastructure? Inheritance is a somewhat strange

Re: [HACKERS] Table Partitioning, Part 1

2005-05-10 Thread Hannu Krosing
On T, 2005-05-10 at 16:31 +0300, Hannu Krosing wrote: On E, 2005-05-09 at 23:30 +0100, Simon Riggs wrote: There are 2 possibly expensive steps: 1. the conversion to AND'ed list of simple clauses (unknown complexity) 2. matching each of simple clauses in the and list with all others

Re: [HACKERS] Table Partitioning, Part 1

2005-05-10 Thread Jim C. Nasby
On Tue, May 10, 2005 at 12:16:17AM +0100, Simon Riggs wrote: On Mon, 2005-05-09 at 18:53 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: 1. Embellish inheritance or separate infrastructure? It seems prudent to avoid building on that foundation, even though we may decide

Re: [HACKERS] Table Partitioning, Part 1

2005-05-10 Thread Tom Lane
Jim C. Nasby [EMAIL PROTECTED] writes: On Tue, May 10, 2005 at 12:16:17AM +0100, Simon Riggs wrote: On Mon, 2005-05-09 at 18:53 -0400, Tom Lane wrote: I disagree. The code is there, it could use work, and what you are basically proposing is to duplicate both the existing work and much of the

Re: [HACKERS] Table Partitioning, Part 1

2005-05-10 Thread Simon Riggs
On Tue, 2005-05-10 at 16:31 +0300, Hannu Krosing wrote: On E, 2005-05-09 at 23:30 +0100, Simon Riggs wrote: ISTM fairly straightforward to produce a similar static plan along the same lines, using Result nodes to implement Partition Elimination. Append Result SeqScan Result

Re: [HACKERS] Table Partitioning, Part 1

2005-05-10 Thread Simon Riggs
On Tue, 2005-05-10 at 16:44 +0300, Hannu Krosing wrote: On T, 2005-05-10 at 16:31 +0300, Hannu Krosing wrote: On E, 2005-05-09 at 23:30 +0100, Simon Riggs wrote: There are 2 possibly expensive steps: 1. the conversion to AND'ed list of simple clauses (unknown complexity) 2.

Re: [HACKERS] Table Partitioning, Part 1

2005-05-10 Thread Simon Riggs
On Tue, 2005-05-10 at 15:01 -0400, Tom Lane wrote: Jim C. Nasby [EMAIL PROTECTED] writes: On Tue, May 10, 2005 at 12:16:17AM +0100, Simon Riggs wrote: On Mon, 2005-05-09 at 18:53 -0400, Tom Lane wrote: I disagree. The code is there, it could use work, and what you are basically proposing

Re: [HACKERS] Table Partitioning, Part 1

2005-05-09 Thread Tom Lane
Simon Riggs [EMAIL PROTECTED] writes: 1. Embellish inheritance or separate infrastructure? It seems prudent to avoid building on that foundation, even though we may decide to use some similar approaches. I disagree. The code is there, it could use work, and what you are basically proposing

<    5   6   7   8   9   10   11   >