Re: [HACKERS] Pluggable Indexes

2009-01-24 Thread Simon Riggs
On Sat, 2009-01-24 at 13:51 +, Simon Riggs wrote: > On Sat, 2009-01-24 at 09:57 +, Simon Riggs wrote: > > > I agree we need an external module and I learned that lesson from the > > earier API proposal you mentioned. The supplied WAL filter plugin was/is > > a valid use for this and, as d

Re: [HACKERS] Pluggable Indexes

2009-01-24 Thread Simon Riggs
On Sat, 2009-01-24 at 09:57 +, Simon Riggs wrote: > I agree we need an external module and I learned that lesson from the > earier API proposal you mentioned. The supplied WAL filter plugin was/is > a valid use for this and, as discussed, is the only practical way of > doing WAL filtering. As

Re: [HACKERS] Pluggable Indexes

2009-01-24 Thread Simon Riggs
On Fri, 2009-01-23 at 16:49 -0500, Tom Lane wrote: > Simon Riggs writes: > > On Fri, 2009-01-23 at 10:33 -0500, Tom Lane wrote: > >> Right, the WAL-record-processing API is not really at issue, since it's > >> been proven internally to the core code. My concern is with the other > >> part, namel

Re: [HACKERS] Pluggable Indexes

2009-01-23 Thread Tom Lane
Simon Riggs writes: > On Fri, 2009-01-23 at 10:33 -0500, Tom Lane wrote: >> Right, the WAL-record-processing API is not really at issue, since it's >> been proven internally to the core code. My concern is with the other >> part, namely exactly how are we going to identify and install additional

Re: [HACKERS] Pluggable Indexes

2009-01-23 Thread Simon Riggs
On Fri, 2009-01-23 at 10:33 -0500, Tom Lane wrote: > Simon Riggs writes: > > On Thu, 2009-01-22 at 18:45 -0500, Tom Lane wrote: > >> There are other recent examples of proposed hooks that in fact > >> failed to be useful because of some oversight or other, and it was > >> not until we insisted on

Re: [HACKERS] Pluggable Indexes

2009-01-23 Thread Tom Lane
Simon Riggs writes: > On Thu, 2009-01-22 at 18:45 -0500, Tom Lane wrote: >> There are other recent examples of proposed hooks that in fact >> failed to be useful because of some oversight or other, and it was >> not until we insisted on seeing a live use of the hooks that this >> became apparent.

Re: [HACKERS] Pluggable Indexes

2009-01-23 Thread Teodor Sigaev
Hmm, IIRC it is based on a monotonically increasing number. It could have been anything. LSN was just a monotonically increasing number that would be available if WAL was implemented first (or in parallel). You are right, but without WAL-logging we would need to implement some kind of sequenc

Re: [HACKERS] Pluggable Indexes

2009-01-23 Thread Simon Riggs
On Thu, 2009-01-22 at 18:45 -0500, Tom Lane wrote: > There are other recent examples of proposed hooks that in fact > failed to be useful because of some oversight or other, and it was > not until we insisted on seeing a live use of the hooks that this > became apparent. (IIRC, one or both of th

Re: [HACKERS] Pluggable Indexes

2009-01-22 Thread Tom Lane
Robert Haas writes: >> Of course, there's no much point in an index that's easily corrupted, so >> I understand the desire to implement WAL too -- I'm just pointing out >> that concurrency could have been developed independently. > Anything's possible with enough work, but having good support in

Re: [HACKERS] Pluggable Indexes

2009-01-22 Thread Robert Haas
> Of course, there's no much point in an index that's easily corrupted, so > I understand the desire to implement WAL too -- I'm just pointing out > that concurrency could have been developed independently. Anything's possible with enough work, but having good support in -core makes it easier and

Re: [HACKERS] Pluggable Indexes

2009-01-22 Thread Alvaro Herrera
Teodor Sigaev wrote: >> What other constraints are there on such non-in-core indexex? Early (2005) >> GIST indexes were very painful in production environments because vacuuming >> them held locks for a *long* time (IIRC, an hour or so on my database) on >> the indexes locking out queries. Was th

Re: [HACKERS] Pluggable Indexes

2009-01-22 Thread Teodor Sigaev
What other constraints are there on such non-in-core indexex? Early (2005) GIST indexes were very painful in production environments because vacuuming them held locks for a *long* time (IIRC, an hour or so on my database) on the indexes locking out queries. Was that just a shortcoming of the imp

Re: [HACKERS] Pluggable Indexes

2009-01-22 Thread tomas
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Wed, Jan 21, 2009 at 10:48:21PM +, Simon Riggs wrote: > > On Thu, 2009-01-22 at 00:29 +0300, Oleg Bartunov wrote: [...] > > Other question, why don't improve GiST to allow support of more indexes ? > > bitmap indexes could be implemented usi

Re: [HACKERS] Pluggable Indexes

2009-01-22 Thread Heikki Linnakangas
Oleg Bartunov wrote: bitmap indexes could be implemented usin g GiST. Huh, how would that work? Bitmap indexes have a very different structure, AFAICS. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)

Re: [HACKERS] Pluggable Indexes

2009-01-21 Thread Simon Riggs
On Thu, 2009-01-22 at 00:00 +, Gregory Stark wrote: > But note that the interface is critical. Yes, it is. The existing rmgr code provides for 5 separate calls that a module needs to implement to make an access method recoverable. btree, hash, gist and gin already implement that API. I hav

Re: [HACKERS] Pluggable Indexes

2009-01-21 Thread Gregory Stark
Josh Berkus writes: > Right. And I'm saying that pluggability is PostgreSQL's main reason for > existence, if you look at our place in the future of databases. So it's worth > paying *some* cost, provided that the cost/benefit ratio works for the > particular patch. I agree that pluggability i

Re: [HACKERS] Pluggable Indexes

2009-01-21 Thread Simon Riggs
On Thu, 2009-01-22 at 00:29 +0300, Oleg Bartunov wrote: > I'm rather keen on Pg extendability, which allowed me and Teodor to > work on many extensions. Yes, first GiST we inherited from early > academic research and was more like a toy. We still have several TODO > items about GiST interface

Re: [HACKERS] Pluggable Indexes

2009-01-21 Thread Oleg Bartunov
On Wed, 21 Jan 2009, Simon Riggs wrote: On Wed, 2009-01-21 at 21:45 +0200, Heikki Linnakangas wrote: Ron Mayer wrote: Early (2005) GIST indexes were very painful in production environments because vacuuming them held locks for a *long* time (IIRC, an hour or so on my database) on the indexes

Re: [HACKERS] Pluggable Indexes

2009-01-21 Thread Simon Riggs
On Wed, 2009-01-21 at 21:45 +0200, Heikki Linnakangas wrote: > Ron Mayer wrote: > > Early (2005) > > GIST indexes were very painful in production environments because vacuuming > > them held locks for a *long* time (IIRC, an hour or so on my database) on > > the indexes locking out queries. Was t

Re: [HACKERS] Pluggable Indexes

2009-01-21 Thread Heikki Linnakangas
Ron Mayer wrote: Early (2005) GIST indexes were very painful in production environments because vacuuming them held locks for a *long* time (IIRC, an hour or so on my database) on the indexes locking out queries. Was that just a shortcoming of the implementation, or was it a side-effect of them

Re: [HACKERS] Pluggable Indexes

2009-01-21 Thread Ron Mayer
Gregory Stark wrote: > Simon Riggs writes: > >> The original design of Postgres allowed pluggable index access methods, >> but that capability has not been brought forward to allow for WAL. This >> patch would bridge that gap. > > Well I think what people do is what GIST did early on -- they jus

Re: [HACKERS] Pluggable Indexes

2009-01-21 Thread Andrew Sullivan
None of this is Any of My Business any more, but On Wed, Jan 21, 2009 at 03:44:15PM +, Simon Riggs wrote: > The patch takes special care to allow calls to the rmgr functions only > from the startup process. The APIs are exactly like the indexams and > *are* called only in specific ways, at sp

Re: [HACKERS] Pluggable Indexes

2009-01-21 Thread Heikki Linnakangas
Gregory Stark wrote: But to do that we need an abstract interface that doesn't depend on internal data structures, not a generic plugin facility that allows the plugin to hijack the whole system. We need something more like indexams which provides a set of call points which do specific functions

Re: [HACKERS] Pluggable Indexes

2009-01-21 Thread Simon Riggs
On Wed, 2009-01-21 at 14:57 +, Gregory Stark wrote: > But to do that we need an abstract interface that doesn't depend on > internal data structures, not a generic plugin facility that allows > the plugin to hijack the whole system. > > We need something more like indexams which provides a se

Re: [HACKERS] Pluggable Indexes

2009-01-21 Thread Gregory Stark
Simon Riggs writes: > The original design of Postgres allowed pluggable index access methods, > but that capability has not been brought forward to allow for WAL. This > patch would bridge that gap. Well I think what people do is what GIST did early on -- they just don't support recoverability u