On 04/23/2015 04:04 PM, Andrew Gierth wrote:
"Joshua" == Joshua D Drake <j...@commandprompt.com> writes:
Joshua> The database dumps fine as long as we don't dump large
Joshua> objects. However, if we try to dump the large objects, FreeBSD
Joshua> will kill pg_dump as it will consume all free memory and
Joshua> swap. With Andrew's help we were able to determine the
Joshua> following:
Joshua> There is a memory cost of about 160 bytes per largeobject.
I may have the exact number here wrong, it was just a quick eyeball of
the data structures (and depends on malloc overheads anyway).
The relevant code is getBlobs in pg_dump.c, which queries the whole of
pg_largeobject_metadata without using a cursor (so the PGresult is
already huge thanks to having >100 million rows), and then mallocs a
BlobInfo array and populates it from the PGresult, also using pg_strdup
for the oid string, owner name, and ACL if any.
I'm surprised this hasn't come up before. I have a client that I
persuaded to convert all their LOs to bytea fields because of problems
with pg_dump handling millions of LOs, and kept them on an older
postgres version until they made that change.
cheers
andrew
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
