Moving to -hackers On Mon, Jun 05, 2006 at 12:32:38AM +0200, Zoltan Boszormenyi wrote: > >I just noticed that psql's unformatted output uses too much > >memory. Is it normal? It seems that psql draws all records > >of a query off the server before it displays or writes the output. > >I would expect this only with formatted output. > > > >Problem is, I have an export that produces 500'000+ records > >which changes frequently. Several (20+) sites run this query > >nightly with different parameters and download it. The SELECTs > >that run in psql -A -t -c '...' may overlap and the query that runs > >in less than 1.5 minutes if it's the only one at the time may take > >3+ hours if ten such queries overlap. The time is mostly spent > >in swapping, all psql processes take up 300+ MB, so the 1GB > >server is brought to its knees quickly, peek swap usage is 1.8 GB. > >I watched the progress in top and the postmaster processes finished > >their work in about half an hour (that would still be acceptable) > >then the psql processes started eating up memory as they read > >the records. > > > >PostgreSQL 8.1.4 was used on RHEL3. > > > >Is there a way to convince psql to use less memory in unformatted > >mode? I know COPY will be able to use arbitrary SELECTs > >but until then I am still stuck with redirecting psql's output. > > The answer it to use SELECT INTO TEMP and then COPY. > Psql will use much less memory that way. But still...
I've been able to verify this on 8.1.4; psql -A -t -c 'SELECT * FROM largetable' > /dev/null results in psql consuming vast quantities of memory. Why is this? ISTM this is a bug... -- Jim C. Nasby, Sr. Engineering Consultant [EMAIL PROTECTED] Pervasive Software http://pervasive.com work: 512-231-6117 vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461 ---------------------------(end of broadcast)--------------------------- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq