Re: [PATCHES] Heap page diagnostic/test functions (v2)
Simon Riggs wrote: New functions to examine the contents of heap pages, as discussed recently on -hackers. These are fully integrated into backend. ... I'll maintain this with immediate fixes/additions as we go up to 8.3 and beyond, to assist review process of various patches that alter page contents. Excellent. I agree with the other comments that these should all be in contrib. The functions need to be very careful to handle invalid data gracefully. For example, this will return bogus data or crash if the offset in the line pointer is corrupt, past the end of page for example: + tuphdr = (HeapTupleHeader) PageGetItem((Page) inter_call_data->page, id); It's particularly important if the page-examining functions are not superuser-only, to avoid vulnerabilities. Even if they are, the functions are for diagnostic purposes and diagnosis is often done on something that's corrupt. Dare I propose deprecating the all the system columns except for ctid in favor of these functions? I guess there's no immediate reason to remove them, but these functions feel like a much better design than tightly integrated system columns. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [PATCHES] Heap page diagnostic/test functions (v2)
On Sat, 2007-04-07 at 18:19 -0400, Bruce Momjian wrote: > Simon Riggs wrote: > > On Sat, 2007-04-07 at 11:53 -0400, Bruce Momjian wrote: > > > > > This looks useful, but shouldn't it be part of /contrib/pgstattuple > > > rather than in the backend? > > > > Well, this was written with a view to it being usable for writing test > > cases that checked the various tuple states as we went. It was > > originally proposed when Tom asked "How will we test HOT?" (concurrent > > psql is the other half of the required base functionality to write > > sensible test cases). > > > > If we see it as a manual test tool only, contrib is OK. But it has to be > > in the backend to be usable in the regression test suite, so thats where > > it was designed to go. > > Well, contrib can have its own regression tests. We can put the HOT > tests in there too. OK. I'll rework it once I've completed the other items on my list. Feel free to have a hack at it if I'm a little slow. I only have today left before I'm away for two weeks - travelling tomorrow. -- Simon Riggs EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [PATCHES] Heap page diagnostic/test functions (v2)
"Simon Riggs" <[EMAIL PROTECTED]> writes: > If we see it as a manual test tool only, contrib is OK. But it has to be > in the backend to be usable in the regression test suite, so thats where > it was designed to go. The core regression tests have depended on some contrib stuff forever, so the above argument holds no water. I'm uncomfortable with putting these things in core because I suspect it's not hard to crash the backend (or worse) by feeding one of them artfully corrupted data. regards, tom lane ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [PATCHES] Heap page diagnostic/test functions (v2)
Simon Riggs wrote: > On Sat, 2007-04-07 at 11:53 -0400, Bruce Momjian wrote: > > > This looks useful, but shouldn't it be part of /contrib/pgstattuple > > rather than in the backend? > > Well, this was written with a view to it being usable for writing test > cases that checked the various tuple states as we went. It was > originally proposed when Tom asked "How will we test HOT?" (concurrent > psql is the other half of the required base functionality to write > sensible test cases). > > If we see it as a manual test tool only, contrib is OK. But it has to be > in the backend to be usable in the regression test suite, so thats where > it was designed to go. Well, contrib can have its own regression tests. We can put the HOT tests in there too. -- Bruce Momjian <[EMAIL PROTECTED]> http://momjian.us EnterpriseDB http://www.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [PATCHES] Heap page diagnostic/test functions (v2)
On Sat, 2007-04-07 at 11:53 -0400, Bruce Momjian wrote: > This looks useful, but shouldn't it be part of /contrib/pgstattuple > rather than in the backend? Well, this was written with a view to it being usable for writing test cases that checked the various tuple states as we went. It was originally proposed when Tom asked "How will we test HOT?" (concurrent psql is the other half of the required base functionality to write sensible test cases). If we see it as a manual test tool only, contrib is OK. But it has to be in the backend to be usable in the regression test suite, so thats where it was designed to go. -- Simon Riggs EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [PATCHES] Heap page diagnostic/test functions (v2)
This looks useful, but shouldn't it be part of /contrib/pgstattuple rather than in the backend? --- Simon Riggs wrote: > New functions to examine the contents of heap pages, as discussed > recently on -hackers. These are fully integrated into backend. > > Designed to be extended for other page layouts/contents. (Heikki has > some similar functions for index pages). > > Docs included, applies cleanly, tests good. > > I'll maintain this with immediate fixes/additions as we go up to 8.3 and > beyond, to assist review process of various patches that alter page > contents. > > -- > Simon Riggs > EnterpriseDB http://www.enterprisedb.com > [ Attachment, skipping... ] > > ---(end of broadcast)--- > TIP 4: Have you searched our list archives? > >http://archives.postgresql.org -- Bruce Momjian <[EMAIL PROTECTED]> http://momjian.us EnterpriseDB http://www.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [PATCHES] Heap page diagnostic/test functions (v2)
Your patch has been added to the PostgreSQL unapplied patches list at: http://momjian.postgresql.org/cgi-bin/pgpatches It will be applied as soon as one of the PostgreSQL committers reviews and approves it. --- Simon Riggs wrote: > On Thu, 2007-03-08 at 15:44 +, Simon Riggs wrote: > > > Docs included > > Just noticed a typo. File mentioned in func.sgml, line 11049 should be > src/include/access/htup.h and not > src/include/storage/bufpage.h > > -- > Simon Riggs > EnterpriseDB http://www.enterprisedb.com > > > > ---(end of broadcast)--- > TIP 3: Have you checked our extensive FAQ? > >http://www.postgresql.org/docs/faq -- Bruce Momjian <[EMAIL PROTECTED]> http://momjian.us EnterpriseDB http://www.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [PATCHES] Heap page diagnostic/test functions (v2)
Your patch has been added to the PostgreSQL unapplied patches list at: http://momjian.postgresql.org/cgi-bin/pgpatches It will be applied as soon as one of the PostgreSQL committers reviews and approves it. --- Simon Riggs wrote: > New functions to examine the contents of heap pages, as discussed > recently on -hackers. These are fully integrated into backend. > > Designed to be extended for other page layouts/contents. (Heikki has > some similar functions for index pages). > > Docs included, applies cleanly, tests good. > > I'll maintain this with immediate fixes/additions as we go up to 8.3 and > beyond, to assist review process of various patches that alter page > contents. > > -- > Simon Riggs > EnterpriseDB http://www.enterprisedb.com > [ Attachment, skipping... ] > > ---(end of broadcast)--- > TIP 4: Have you searched our list archives? > >http://archives.postgresql.org -- Bruce Momjian <[EMAIL PROTECTED]> http://momjian.us EnterpriseDB http://www.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [PATCHES] Heap page diagnostic/test functions (v2)
On Thu, 2007-03-08 at 15:44 +, Simon Riggs wrote: > Docs included Just noticed a typo. File mentioned in func.sgml, line 11049 should be src/include/access/htup.h and not src/include/storage/bufpage.h -- Simon Riggs EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
[PATCHES] Heap page diagnostic/test functions (v2)
New functions to examine the contents of heap pages, as discussed
recently on -hackers. These are fully integrated into backend.
Designed to be extended for other page layouts/contents. (Heikki has
some similar functions for index pages).
Docs included, applies cleanly, tests good.
I'll maintain this with immediate fixes/additions as we go up to 8.3 and
beyond, to assist review process of various patches that alter page
contents.
--
Simon Riggs
EnterpriseDB http://www.enterprisedb.com
Index: doc/src/sgml/func.sgml
===
RCS file: /projects/cvsroot/pgsql/doc/src/sgml/func.sgml,v
retrieving revision 1.369
diff -c -r1.369 func.sgml
*** doc/src/sgml/func.sgml 20 Feb 2007 19:59:04 - 1.369
--- doc/src/sgml/func.sgml 8 Mar 2007 15:34:26 -
***
*** 10854,10859
--- 10854,11064
at session end, even if the client disconnects ungracefully.)
+
+ Diagnostic Functions
+
+
+ Name Return Type Description
+
+
+
+
+
+
+ heap_get_raw_page(RelName text, blockId int)
+
+bytea(BLCKSZ)
+Get binary copy of a shared buffer page
+
+
+
+
+ raw_page_header(rawpage bytea)
+
+record
+Show page header fields of a raw page
+
+
+
+
+ raw_page_heap_tuple_headers(rawpage bytea)
+
+setof record
+Show all tuple headers on heap raw page
+
+
+
+
+
+
+ heap_get_raw_page
+
+
+ heap_get_raw_page reads one block of the named table and
+ returns a copy as a bytea field. This allows a single time-consistent
+ copy of the block to be made for diagnostic purposes. The block
+ is read from shared memory, so the block is already known Valid
+ when it is read in from disk. Use of this functions is restricted to
+ superusers.
+
+
+
+ raw_page_header
+
+
+ raw_page_header shows fields which are common to all types
+ of PostgreSQL page, so can be used with both heap and index pages. Use of
+ this function is not restricted to superusers, though a raw page must be
+ provided as input, which can only be gained by a superuser executing
+ heap_get_raw_page.
+
+ postgres=# select * from raw_page_header(heap_get_raw_page('foo',0));
+
+lsn| tli | pd_lower | pd_upper | pd_special | version
+ --+-+--+--++-
+ 0/47FC08 | 1 | 420 | 5392 | 8192 | 4
+ (1 row)
+
+ The fields are defined in src/include/storage/bufpage.h
+
+
+
+ raw_page_heap_tuple_headers
+
+
+ raw_page_heap_tuple_headers shows all line pointers on a heap page.
+ Most or all of these will be filled with tuples and, when present,
+ tuple headers are shown. All tuples are shown, whether or not
+ the tuples were visible to an MVCC snapshot at the time the raw page
+ was copied. Use of this function is not restricted to superusers, though
+ a raw page must be provided as input, which can only be gained by a
+ superuser executing heap_get_raw_page.
+
+ postgres=# select * from raw_page_heap_tuple_headers(heap_get_raw_page('foo',0)) limit 1;
+
+ id | ok | lpoff | len | tctid | xmn | cb | xmx | cx | cid | natts | toid | info | hdrsz
+ -++---+-+-+--++--++-+---+--+---+---
+1 | t | 5420 | 28 | (0,1) | 2223 | c |0 | i | 0 | 1 | | 10496 |24
+2 | t | 5364 | 28 | (0,2) | 2464 ||0 | i | 0 | 1 | | 10240 |24
+3 | f | | | | || || | | | |
+4 | t | 8164 | 28 | (0,2) | 602 | c | 2464 || 0 | 1 | | 256 |24
+5 | t | 8136 | 28 | (0,5) | 602 | c |0 | i | 0 | 1 | | 2304 |24
+
+ The fields shown are defined in detail in
+ src/include/storage/bufpage.h, some of which are formatted
+ for easier interpretation as shown in the table below.
+
+
+
+ Heap Tuple Header Fields
+
+
+ Field Type Description
+
+
+
+
+
+id
+int
+Line pointer
+
+
+ok
+boolean
+is item and tuple valid?
+
+
+lpoff
+int
+Byte offset within page to start of tuple
+
+
+len
+int
+Length of tuple
+
+
+tctid
+text
+tuple identifier (tid) in (blockid, itemid) format
+
+
+xmn
+int
+xmin of tuple
+
+
+cn
+char(1)
+xmn commit status:
+ c = known committed,
+ i = known invalid,
+ blank = unknown
+
+
+
+xmx
+int
+xmax of tuple or 0 if not set
+
