Re: [HACKERS] Adding probes for smgr
(2012/07/29 12:14), Tom Lane wrote: > Peter Geoghegan writes: >> On 28 July 2012 17:15, Tom Lane wrote: >>> IMV smgr is pretty vestigial. I wouldn't recommend loading more >>> functionality onto that layer, because it's as likely as not that >>> we'll just get rid of it someday. > >> Agreed. I recently found myself reading a paper written by Stonebraker >> back in the Berkeley days: >> http://dislab2.hufs.ac.kr/dislab/seminar/2007/ERL-M87-06.pdf >> This paper appears to have been published in about 1988, and it shows. >> It's fairly obvious from reading the opening paragraph that the >> original rationale for the design of the storage manager doesn't hold >> these days. Of course, it's also obvious from reading the code, since >> for example there is only one storage manager module. > > Yeah. There were actually two storage managers in what we inherited > from Berkeley, but we soon got rid of the other one as being useless. > (IIRC it was meant for magnetic-core memory ... anybody seen any of > that lately?) I remember that I had found mm.c when I started looking around the storage manager code, maybe a decade ago. It seemed very interesting, but I've never seen "magnetic-core memory" itself. :) Anyway, I'm realizing that it's reasonable to add new probes into the md module at this time. Thanks for the comments. > > I think basically what happened since then is that the functionality > Stonebraker et al imagined as being in per-storage-manager code all > migrated into the kernel device drivers, or even down into the hardware > itself. (SSDs are *way* smarter than the average '80s storage device, > and even those were an order of magnitude smarter than what they'd been > ten years previously. I used to do device drivers back in the 80's...) > There's no longer any good reason to have anything but md.c, which isn't > so much a "magnetic disk" interface as an "interface to something that > has a Unix block device driver". BTW, I'm still interested in keeping the storage manager architecture "theoretically" pluggable. From my point of view, current pluggable architecture was designed to serve for different storage manager with by-passing the posix file system API. I agree with that most of recent storage devices can be managed under ordinally file systems, ext3/ext4 or xfs for examples. However, I'm still curious to see new project which will intend to extend storage manager to earn more performance for some special storage devices by by-passing ordinally file system API. For example, Fusion-IO is offering dedicated API for their PCI-Express Flash storage, and I guess the database performace could tremendously benefit from it. (I accept that it's just my curious though. :) It's just a possible option, but keeping the storage architecture pluggable does make sense to me even if the community could give only one storage manager. Regards, > >> This state of affairs sort of reminds me of mcxt.c . The struct >> MemoryContextData is described as an "abstract type" that can have >> multiple implementations, despite the fact that since 2000 (and >> perhaps earlier), the underlying type is invariably AllocSetContext. I >> never investigated if that indirection still needs to exist, but I >> suspect that it too is a candidate for refactoring. Do you agree? > > Meh. Having invented the MemoryContext interface, I am probably not > the best-qualified person to evaluate it objectively. The original > thought was that we might have (a) a context type that could allocate > storage in shared memory, and/or (b) a context type that could provide > better allocation speed at a loss of storage efficiency (eg, lose the > ability to pfree individual chunks). Case (a) has never become > practical given the inability of SysV-style shared memory to expand at > all. I don't know if that might change when/if we switch to some other > shmem API. The idea of a different allocation strategy for some usages > still seems like something we'll want to do someday, though. > > regards, tom lane > -- Satoshi Nagayasu Uptime Technologies, LLC. http://www.uptime.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding probes for smgr
Peter Geoghegan writes: > On 28 July 2012 17:15, Tom Lane wrote: >> IMV smgr is pretty vestigial. I wouldn't recommend loading more >> functionality onto that layer, because it's as likely as not that >> we'll just get rid of it someday. > Agreed. I recently found myself reading a paper written by Stonebraker > back in the Berkeley days: > http://dislab2.hufs.ac.kr/dislab/seminar/2007/ERL-M87-06.pdf > This paper appears to have been published in about 1988, and it shows. > It's fairly obvious from reading the opening paragraph that the > original rationale for the design of the storage manager doesn't hold > these days. Of course, it's also obvious from reading the code, since > for example there is only one storage manager module. Yeah. There were actually two storage managers in what we inherited from Berkeley, but we soon got rid of the other one as being useless. (IIRC it was meant for magnetic-core memory ... anybody seen any of that lately?) I think basically what happened since then is that the functionality Stonebraker et al imagined as being in per-storage-manager code all migrated into the kernel device drivers, or even down into the hardware itself. (SSDs are *way* smarter than the average '80s storage device, and even those were an order of magnitude smarter than what they'd been ten years previously. I used to do device drivers back in the 80's...) There's no longer any good reason to have anything but md.c, which isn't so much a "magnetic disk" interface as an "interface to something that has a Unix block device driver". > This state of affairs sort of reminds me of mcxt.c . The struct > MemoryContextData is described as an "abstract type" that can have > multiple implementations, despite the fact that since 2000 (and > perhaps earlier), the underlying type is invariably AllocSetContext. I > never investigated if that indirection still needs to exist, but I > suspect that it too is a candidate for refactoring. Do you agree? Meh. Having invented the MemoryContext interface, I am probably not the best-qualified person to evaluate it objectively. The original thought was that we might have (a) a context type that could allocate storage in shared memory, and/or (b) a context type that could provide better allocation speed at a loss of storage efficiency (eg, lose the ability to pfree individual chunks). Case (a) has never become practical given the inability of SysV-style shared memory to expand at all. I don't know if that might change when/if we switch to some other shmem API. The idea of a different allocation strategy for some usages still seems like something we'll want to do someday, though. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding probes for smgr
On 28 July 2012 17:15, Tom Lane wrote: > IMV smgr is pretty vestigial. I wouldn't recommend loading more > functionality onto that layer, because it's as likely as not that > we'll just get rid of it someday. Agreed. I recently found myself reading a paper written by Stonebraker back in the Berkeley days: http://dislab2.hufs.ac.kr/dislab/seminar/2007/ERL-M87-06.pdf This paper appears to have been published in about 1988, and it shows. It's fairly obvious from reading the opening paragraph that the original rationale for the design of the storage manager doesn't hold these days. Of course, it's also obvious from reading the code, since for example there is only one storage manager module. This state of affairs sort of reminds me of mcxt.c . The struct MemoryContextData is described as an "abstract type" that can have multiple implementations, despite the fact that since 2000 (and perhaps earlier), the underlying type is invariably AllocSetContext. I never investigated if that indirection still needs to exist, but I suspect that it too is a candidate for refactoring. Do you agree? Incidentally, you might consider refreshing these remarks in the smgr README: "In Berkeley Postgres each relation was tagged with the ID of the storage manager to use for it. This is gone. It would be more reasonable to associate storage managers with tablespaces (a feature not present as this text is being written, but one likely to emerge soon)." -- Peter Geoghegan http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training and Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Adding probes for smgr
Satoshi Nagayasu writes: > Hi, > I'm thinking of adding new probes to trace smgr activities. > In this implementation, I just found that md.c has its own probes > within it, but I'm wondering why we do not have those probes > within the generic smgr routines itself. IMV smgr is pretty vestigial. I wouldn't recommend loading more functionality onto that layer, because it's as likely as not that we'll just get rid of it someday. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Adding probes for smgr
Hi, I'm thinking of adding new probes to trace smgr activities. In this implementation, I just found that md.c has its own probes within it, but I'm wondering why we do not have those probes within the generic smgr routines itself. Which would be a better choice? Any ideas or comments? Regards, -- Satoshi Nagayasu Uptime Technologies, LLC. http://www.uptime.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers