On 12/13/2013 08:40 PM, Alvaro Herrera wrote:
Heikki Linnakangas escribió:
I haven't been following this thread in detail, but would it help if
we implemented a scheme to reduce (auto)vacuum's memory usage? Such
schemes have been discussed in the past, packing the list of dead
items more tightly.
Well, it would help some, but it wouldn't eliminate the problem
completely. Autovacuum scales its memory usage based on the size of the
table. There will always be a table so gigantic that a maximum
allocated memory is to be expected; and DBAs will need a way to limit
the memory consumption even if we pack dead items more densely.
I was playing with keeping item pointers for each page in a bitmapset.
This was pretty neat and used a lot less memory than currently, except
that I needed to allocate a large chunk of memory and then have
bitmapsets use words within that large allocation space. It turned out
to be too ugly so I abandoned it. With the "varbit encoding" thingy in
the recent GIN patchset, maybe it would be workable.
The varbyte encoding is actually a very poor fit for vacuum. Vacuum
needs fast random access into the array when scanning indexes, and the
varbyte encoded item pointer lists used in gin don't allow that.
I couldn't find it in the archives now, but when we last discussed this,
Tom suggested that we divide the large chunk of memory that vacuum
allocates into two parts. The first part grows from the bottom up, and
the second part from top down, until there is no free space in the
middle anymore. For each heap page, there is one entry in the first
part, with the block number, and a pointer to an entry in the second
part. In the second part, there's a list of offset numbers on that page
(or a bitmap).
Another idea: Store only the least significant 20 bits the block number
of each item pointer, and use the remaining 12 bits for the offset
number. So each item pointer is stored as a single 32 bit integer. For
the top 12 bits of the block number, build a separate lookup table of
4096 entries, indexed by the top bits. Each entry in the lookup table
points to the beginning and end index in the main array where the
entries for that page range is stored. That would reduce the memory
usage by about 1/3, which isn't as good as the bitmap method when there
is a lot of dead tuples same pages, but would probably be a smaller patch.
- Heikki
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
