Re: [HACKERS] Surprising dead_tuple_count from pgstattuple

2010-08-10 Thread Gordon Shannon
After much code reading, testing, and using the extremely handy pageinspect contrib to look at pages, here's what I believe is happening. I am not attempting to describe every possible scenario, only this one test path. Following my short test scenario above... - Inserted rows get line pointers

Re: [HACKERS] Surprising dead_tuple_count from pgstattuple

2010-08-10 Thread Tom Lane
Gordon Shannon gordo...@gmail.com writes: - Because my table has no indexes, lazy_scan_heap calls lazy_vacuum_page directly for each block, and reports the variable tups_vacuumed (removed 200 row versions in 2 pages). However, tups_vacuumed is computed without counting the 100 LP_DEAD tuples,

Re: [HACKERS] Surprising dead_tuple_count from pgstattuple

2010-08-09 Thread Simon Riggs
On Sat, 2010-08-07 at 16:11 -0700, Gordon Shannon wrote: So, I guess my real question here is, what happened to the missing 100 items? If it was HOT prune, can anyone summarize what that does? Itagaki already explained that the second DELETE would have removed the 100 dead rows you consider

Re: [HACKERS] Surprising dead_tuple_count from pgstattuple

2010-08-09 Thread Robert Haas
On Mon, Aug 9, 2010 at 1:50 PM, Simon Riggs si...@2ndquadrant.com wrote: On Sat, 2010-08-07 at 16:11 -0700, Gordon Shannon wrote: So, I guess my real question here is, what happened to the missing 100 items?  If it was HOT prune, can anyone summarize what that does? Itagaki already explained

Re: [HACKERS] Surprising dead_tuple_count from pgstattuple

2010-08-09 Thread Heikki Linnakangas
On 09/08/10 21:21, Robert Haas wrote: On Mon, Aug 9, 2010 at 1:50 PM, Simon Riggssi...@2ndquadrant.com wrote: Any SQL statement that reads a block can do HOT pruning, if the block is otherwise unlocked. Where does heap_page_prune() get called from in the DELETE path? heapgetpage() --

Re: [HACKERS] Surprising dead_tuple_count from pgstattuple

2010-08-09 Thread Robert Haas
On Mon, Aug 9, 2010 at 2:23 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 09/08/10 21:21, Robert Haas wrote: On Mon, Aug 9, 2010 at 1:50 PM, Simon Riggssi...@2ndquadrant.com  wrote: Any SQL statement that reads a block can do HOT pruning, if the block is otherwise

Re: [HACKERS] Surprising dead_tuple_count from pgstattuple

2010-08-09 Thread Itagaki Takahiro
2010/8/10 Simon Riggs si...@2ndquadrant.com: Any SQL statement that reads a block can do HOT pruning, if the block is otherwise unlocked. We use the term HOT for two features: 1. HOT updates: Avoiding index updates when keys are not modified. 2. HOT pruning: Removing tuple bodies, that

Re: [HACKERS] Surprising dead_tuple_count from pgstattuple

2010-08-09 Thread Simon Riggs
On Tue, 2010-08-10 at 07:43 +0900, Itagaki Takahiro wrote: 2010/8/10 Simon Riggs si...@2ndquadrant.com: Any SQL statement that reads a block can do HOT pruning, if the block is otherwise unlocked. We use the term HOT for two features: 1. HOT updates: Avoiding index updates when keys are

Re: [HACKERS] Surprising dead_tuple_count from pgstattuple

2010-08-08 Thread Kevin Grittner
Gordon Shannon wrote: If it was HOT prune, can anyone summarize what that does? Get a copy of the PostgreSQL source, and read this file: src/backend/access/heap/README.HOT -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your

Re: [HACKERS] Surprising dead_tuple_count from pgstattuple

2010-08-07 Thread Heikki Linnakangas
On 07/08/10 07:43, Gordon Shannon wrote: Regarding HOT prune, I never did any updates, so I think there couldn't be any HOT tuples. Or does HOT prune do more than that? Yes, HOT will also prune away DELETEd tuples. It will leave behind a dead line pointer, so it won't stop the table from

Re: [HACKERS] Surprising dead_tuple_count from pgstattuple

2010-08-07 Thread Gordon Shannon
I think this simple test highlights the question well. -- create temporary table t(x int) with (autovacuum_enabled=off); insert into t select x from generate_series(1,1,1)x; vacuum verbose t; select dead_tuple_count from pgstattuple('t');-- 0 delete from t where x = 100;

[HACKERS] Surprising dead_tuple_count from pgstattuple

2010-08-06 Thread Gordon Shannon
This is an expansion of the question I posed in this thread: http://postgresql.1045698.n5.nabble.com/Need-help-understanding-vacuum-verbose-output-tp2265895p2266912.html I am framing the question here in relation to pgstattuple. Running 8.4.4 on Centos. I have a table T with 5,063,463 rows.

Re: [HACKERS] Surprising dead_tuple_count from pgstattuple

2010-08-06 Thread Itagaki Takahiro
2010/8/7 Gordon Shannon gordo...@gmail.com: 1. I delete 10,000 rows. pgstattuple.dead_tuple_count - 1 2. I delete 15,000 more rows. pgstattuple.dead_tuple_count - 15000 ?? pgstattuple now appears to count the earlier 10K deleted tuples as no longer dead, but free space. I think it's

Re: [HACKERS] Surprising dead_tuple_count from pgstattuple

2010-08-06 Thread Robert Haas
On Fri, Aug 6, 2010 at 9:11 PM, Itagaki Takahiro itagaki.takah...@gmail.com wrote: 2010/8/7 Gordon Shannon gordo...@gmail.com: 1. I delete 10,000 rows. pgstattuple.dead_tuple_count - 1 2. I delete 15,000 more rows. pgstattuple.dead_tuple_count - 15000 ?? pgstattuple now appears to

Re: [HACKERS] Surprising dead_tuple_count from pgstattuple

2010-08-06 Thread Gordon Shannon
Robert Haas wrote: My thought would be is autovacuum running in the background in between these commands?. That's a good thought, but no, autovacuum_vacuum_scale_factor is set to 0.2, meaning that over 1 million dead tuples are necessary for autovacuum. Besides, if autovacuum had run, I