Re: [GENERAL] Memory issue on FreeBSD

2012-11-05 Thread Frank Broniewski
Hi, I just add the different memory values together (minus the buffers). Usually this sums up (+/-) to the installed memory size, at least on my other machines. I found a thread similar to my problem here [1], but no solution. I don't mind top showing false values, but if there's a larger pro

Re: [GENERAL] Memory issue on FreeBSD

2012-11-05 Thread Achilleas Mantzios
How do you measure that smth is missing from top? What values do you add? I am currently running 8.3 but we shouldn't be so far apart top-wise. What is the reading under SIZE and RES in top for all postgresql processes? Take note that shared mem should be recorded for each and every postmaster run

Re: [GENERAL] Memory issue on FreeBSD

2012-11-05 Thread Frank Broniewski
Hi, thank you for your feedback. I had a look at those commands and their output, especially in conjunction with the SEGSZ value from icps -am Here's an example output: # ipcs -am Shared Memory: T ID KEY MODEOWNERGROUPCREATOR CGROUP NATTCHSE

Re: [GENERAL] Memory issue on FreeBSD

2012-11-05 Thread Achilleas Mantzios
(scrap my previous internal email (hence fake) address this one is correct : sorry for that) You can stop pgsql, start it and then watch out for the increase in SEGSZ values. I pretty much think they are in bytes. I am pretty confident that this value depicts the shared_buffers size in bytes.

Re: [GENERAL] Memory issue on FreeBSD

2012-11-05 Thread Achilleas Mantzios
ipcs in FreeBSD is a little ... tricky. ipcs -M ipcs -m ipcs -am could be your friends On Δευ 05 Νοε 2012 11:22:46 Frank Broniewski wrote: > Hi, > > I am running a PostgreSQL server on FreeBSD. The system has 32GB memory. > Usually I use top to examine the memory usage of the system. After a

[GENERAL] Memory issue on FreeBSD

2012-11-05 Thread Frank Broniewski
Hi, I am running a PostgreSQL server on FreeBSD. The system has 32GB memory. Usually I use top to examine the memory usage of the system. After a while, a part, approximately 5GB, vanish from top, so that the memory rounds up to 27GB. After restarting PostgreSQL, I have all 32GB again availa

Re: [GENERAL] Memory issues

2012-10-14 Thread Shiran Kleiderman
Hi This is the output of meminfo when the system is under some stress. Thanks cif@ip-10-194-167-240:/tmp$ cat /proc/meminfo MemTotal:7629508 kB MemFree: 37820 kB Buffers:2108 kB Cached: 5500200 kB SwapCached: 332 kB Active: 4172020 kB Inacti

Re: [GENERAL] Memory issues

2012-10-14 Thread Shiran Kleiderman
Hi I've returned the memory configs to the default, erased data from my db and am testing the system again. This is the output of *cat /proc/meminfo* Thanks root@ip-10-194-167-240:~# cat /proc/meminfo MemTotal:7629508 kB MemFree: 170368 kB Buffers: 10272 kB Cached:

Re: [GENERAL] Memory issues

2012-09-27 Thread Andres Freund
On Monday, September 24, 2012 08:45:06 AM Shiran Kleiderman wrote: > Hi, > I'm using and Amazon ec2 instance with the following spec and the > application that I'm running uses a postgres DB 9.1. > The app has 3 main cron jobs. > > *Ubuntu 12, High-Memory Extra Large Instance > 17.1 GB of memory >

Re: [GENERAL] Memory issues

2012-09-26 Thread Shiran Kleiderman
Hi Thanks again. Right now, this is *free -m and ps aux* and non of the crons can run - can't allocate memory. cif@domU-12-31-39-08-06-20:~$ free -m total used free sharedbuffers cached Mem: 17079 12051 5028 0270 9578 -

Re: [GENERAL] Memory issues

2012-09-26 Thread Scott Marlowe
On Tue, Sep 25, 2012 at 7:00 PM, Shiran Kleiderman wrote: > > Hi > Thanks for your answer. > I understood that the server is ok memory wise. > What can I check on the client side or the DB queries? Well you're connecting to localhost so I'd expect you to show a memory issue in free I'm not seeing

Re: [GENERAL] Memory issues

2012-09-25 Thread Scott Marlowe
On Mon, Sep 24, 2012 at 12:45 AM, Shiran Kleiderman wrote: > > > Hi, > I'm using and Amazon ec2 instance with the following spec and the > application that I'm running uses a postgres DB 9.1. > The app has 3 main cron jobs. > > Ubuntu 12, High-Memory Extra Large Instance > 17.1 GB of memory > 6.5

Re: [GENERAL] Memory issues

2012-09-25 Thread Shiran Kleiderman
Hi Thanks for your answer. I understood that the server is ok memory wise. What can I check on the client side or the DB queries? Thank u. On Wed, Sep 26, 2012 at 2:56 AM, Scott Marlowe wrote: > On Mon, Sep 24, 2012 at 12:45 AM, Shiran Kleiderman > wrote: > > > > > > Hi, > > I'm using and Amazon

[GENERAL] Memory issues

2012-09-24 Thread Shiran Kleiderman
Hi, I'm using and Amazon ec2 instance with the following spec and the application that I'm running uses a postgres DB 9.1. The app has 3 main cron jobs. *Ubuntu 12, High-Memory Extra Large Instance 17.1 GB of memory 6.5 EC2 Compute Units (2 virtual cores with 3.25 EC2 Compute Units each) 420 GB of

Re: [GENERAL] Memory error in user-defined aggregation function

2012-08-07 Thread Adriaan Joubert
Hi, Finally got this running under the debugger and figured out what is going on. I had been under the impression that if (PG_ARGISNULL(0)) PG_RETURN_NULL(); state = (quartile_state *) PG_GETARG_POINTER(0); would ensure that state was never a null pointer. How

Re: [GENERAL] Memory error in user-defined aggregation function

2012-08-07 Thread Tom Lane
Adriaan Joubert writes: > I've implemented an aggregation function to compute quartiles in C > borrowing liberally from orafce code. I uses this code in a windowing > context and it worked fine until today - and I'm not sure what > changed. This is on 9.1.2 and I have also tried it on 9.1.4. Hm,

[GENERAL] Memory error in user-defined aggregation function

2012-08-07 Thread Adriaan Joubert
Hi, I've implemented an aggregation function to compute quartiles in C borrowing liberally from orafce code. I uses this code in a windowing context and it worked fine until today - and I'm not sure what changed. This is on 9.1.2 and I have also tried it on 9.1.4. What I have determined so far (b

Re: [GENERAL] Memory Overcommit

2012-06-07 Thread Tom Lane
Andy Chambers writes: > We've just run into the dreaded "OOM Killer". I see that on Linux >> 2.6, it's recommended to turn off memory overcommit. I'm trying to > understand the implications of doing this. The interweb says this > means that forking servers can't make use of "copy on write" > se

[GENERAL] Memory Overcommit

2012-06-07 Thread Andy Chambers
Hi All, We've just run into the dreaded "OOM Killer". I see that on Linux >2.6, it's recommended to turn off memory overcommit. I'm trying to understand the implications of doing this. The interweb says this means that forking servers can't make use of "copy on write" semantics. Is this true?

Re: [GENERAL] Memory Management in pqlib, Garbage Collection support

2012-05-03 Thread Alexander Reichstadt
(Sorry, I meant libpq). Actually it's finalize in Objective-C as well. PGSQLKit is the ObjC wrapper framework for libpq. I was confused by what I had learnt about GC, being it can't mix with non-GC. What the docu didn't mention in the places I read --or at least not so that it stuck-- was that i

Re: [GENERAL] Memory Management in pqlib, Garbage Collection support

2012-05-03 Thread Alban Hertroys
On 3 May 2012 09:39, Alexander Reichstadt wrote: > Thanks, that's answering my question. In Objective-C as well as many other I notice that you're talking about pqlib instead of libpq. Perhaps pqlib is an Obj-C wrapper around libpq that most of us just don't know about? Obj-C is not a particularl

Re: [GENERAL] Memory Management in pqlib, Garbage Collection support

2012-05-03 Thread Simon Riggs
On Thu, May 3, 2012 at 8:56 AM, Magnus Hagander wrote: > On Thu, May 3, 2012 at 9:39 AM, Alexander Reichstadt wrote: >> Thanks, that's answering my question. In Objective-C as well as many other >> languages there is the feature to turn on Garbage Collection. It's a >> separate thread that scans

Re: [GENERAL] Memory Management in pqlib, Garbage Collection support

2012-05-03 Thread Peter Bex
On Thu, May 03, 2012 at 09:39:29AM +0200, Alexander Reichstadt wrote: > Thanks, that's answering my question. In Objective-C as well as many other > languages there is the feature to turn on Garbage Collection. It's a separate > thread that scans memory for strong pointers, their source and origi

Re: [GENERAL] Memory Management in pqlib, Garbage Collection support

2012-05-03 Thread Magnus Hagander
On Thu, May 3, 2012 at 9:39 AM, Alexander Reichstadt wrote: > Thanks, that's answering my question. In Objective-C as well as many other > languages there is the feature to turn on Garbage Collection. It's a > separate thread that scans memory for strong pointers, their source and > origin and "va

Re: [GENERAL] Memory Management in pqlib, Garbage Collection support

2012-05-03 Thread Alexander Reichstadt
Thanks, that's answering my question. In Objective-C as well as many other languages there is the feature to turn on Garbage Collection. It's a separate thread that scans memory for strong pointers, their source and origin and "vacuums" memory so to not have any leaks. Anything unreferenced and

Re: [GENERAL] Memory Management in pqlib, Garbage Collection support

2012-05-03 Thread John R Pierce
On 05/03/12 12:08 AM, Alexander Reichstadt wrote: since I got no answer so far I searched through the docu again. I searched for GC as well as Garbage, and all garbage refers to is with regard to vacuuming a database. But my question refers to wether or not memory management is with garbage co

Re: [GENERAL] Memory Management in pqlib, Garbage Collection support

2012-05-03 Thread Peter Bex
On Thu, May 03, 2012 at 09:08:53AM +0200, Alexander Reichstadt wrote: > > Hi, > > since I got no answer so far I searched through the docu again. I searched > for GC as well as Garbage, and all garbage refers to is with regard to > vacuuming a database. But my question refers to wether or not m

[GENERAL] Memory Management in pqlib, Garbage Collection support

2012-05-03 Thread Alexander Reichstadt
Hi, since I got no answer so far I searched through the docu again. I searched for GC as well as Garbage, and all garbage refers to is with regard to vacuuming a database. But my question refers to wether or not memory management is with garbage collection supported or not. When I try to link

Re: [GENERAL] Memory usage and configuration settings

2012-03-05 Thread Merlin Moncure
On Mon, Mar 5, 2012 at 6:37 AM, Mike C wrote: > Hi, > > I have been using table 17-2, Postgres Shared Memory Usage > (http://www.postgresql.org/docs/9.1/interactive/kernel-resources.html) > to calculate approximately how much memory the server will use. I'm > using Postgres 9.1 on a Linux 2.6 (RHE

Re: [GENERAL] Memory usage and configuration settings

2012-03-05 Thread Tom Lane
Mike C writes: > Ok, that makes sense. With regards to work_mem, am I right in thinking > the child processes only allocate enough memory to meet the task at > hand, rather than the full 16M specified in the config file? They only allocate what's needed ... but you have to keep in mind that work_

Re: [GENERAL] Memory usage and configuration settings

2012-03-05 Thread Mike C
On Mon, Mar 5, 2012 at 4:04 PM, Tom Lane wrote: > Mike C writes: >> I have been using table 17-2, Postgres Shared Memory Usage >> (http://www.postgresql.org/docs/9.1/interactive/kernel-resources.html) >> to calculate approximately how much memory the server will use. I'm >> using Postgres 9.1 on

Re: [GENERAL] Memory usage and configuration settings

2012-03-05 Thread Tom Lane
Mike C writes: > I have been using table 17-2, Postgres Shared Memory Usage > (http://www.postgresql.org/docs/9.1/interactive/kernel-resources.html) > to calculate approximately how much memory the server will use. I'm > using Postgres 9.1 on a Linux 2.6 (RHEL 6) 64bit system, with 8GB RAM. > Data

[GENERAL] Memory usage and configuration settings

2012-03-05 Thread Mike C
Hi, I have been using table 17-2, Postgres Shared Memory Usage (http://www.postgresql.org/docs/9.1/interactive/kernel-resources.html) to calculate approximately how much memory the server will use. I'm using Postgres 9.1 on a Linux 2.6 (RHEL 6) 64bit system, with 8GB RAM. Database is approximately

Re: [GENERAL] Memory tuning Postgres

2012-02-15 Thread Robert James
On 2/15/12, Tomas Vondra wrote: > On 15 Únor 2012, 15:20, Robert James wrote: >> What parameters should I change to use the server best? What are good >> starting points or them? What type of performance increase should I >> see? ... > But you haven't > mentioned which version of PostgreSQL is us

Re: [GENERAL] Memory tuning Postgres

2012-02-15 Thread Tomas Vondra
On 15 Únor 2012, 15:20, Robert James wrote: > I have a 4 core, 4 GB server dedicated to running Postgres (only other > thing on it are monitoring, backup, and maintenance programs). It > runs about 5 databases, backing up an app, mainly ORM queries, but > some reporting and more complicated SQL JO

[GENERAL] Memory tuning Postgres

2012-02-15 Thread Robert James
I have a 4 core, 4 GB server dedicated to running Postgres (only other thing on it are monitoring, backup, and maintenance programs). It runs about 5 databases, backing up an app, mainly ORM queries, but some reporting and more complicated SQL JOINs as well. I'm currently using the out-of-the box

Re: [GENERAL] memory leak

2012-01-10 Thread Michael P. Soulier
On 10/01/12 Simon Riggs said: > You're missing 2 PQclear() calls on success. > > http://www.postgresql.org/docs/devel/static/libpq-exec.html#LIBPQ-EXEC-MAIN Ah, thanks. Diffing db.c to db.c@@/main/soulierm_MASTeleworker_dev1/3 --- db.c@@/main/soulierm_MASTeleworker_dev1/3 2011-08-10 07:09:27.

Re: [GENERAL] memory leak

2012-01-10 Thread Simon Riggs
On Tue, Jan 10, 2012 at 6:48 PM, Michael P. Soulier wrote: >    res = PQexec(conn, "BEGIN"); >    if (PQresultStatus(res) != PGRES_COMMAND_OK) >    { >        fprintf(stderr, "DB: BEGIN command failed: %s", PQerrorMessage(conn)); >        PQclear(res); >        exit_nicely(conn); >    } > >    re

[GENERAL] memory leak

2012-01-10 Thread Michael P. Soulier
Hi, I've written a small multi-threaded C program using libpq, and valgrind is reporting a memory leak. 2012-01-10 13:45:07.263078500 ==12695== 608 bytes in 4 blocks are definitely lost in loss record 19 of 22 2012-01-10 13:45:07.263097500 ==12695==at 0x4005B83: malloc (vg_replace_malloc.c:

Re: [GENERAL] Memory Issue

2011-11-04 Thread Ioana Danes
Cc: PostgreSQL General Sent: Thursday, November 3, 2011 10:30:27 AM Subject: Re: [GENERAL] Memory Issue On Thu, Nov 3, 2011 at 7:34 AM, Ioana Danes wrote: > After another half an hour almost the entire swap is used and the system > performs really bad 100 TPS or lower. > It never ru

Re: [GENERAL] Memory Issue

2011-11-03 Thread Ioana Danes
- Original Message - From: Scott Marlowe To: Ioana Danes Cc: PostgreSQL General Sent: Thursday, November 3, 2011 10:30:27 AM Subject: Re: [GENERAL] Memory Issue On Thu, Nov 3, 2011 at 7:34 AM, Ioana Danes wrote: > After another half an hour almost the entire swap is used and

Re: [GENERAL] Memory Issue

2011-11-03 Thread Scott Marlowe
On Thu, Nov 3, 2011 at 7:34 AM, Ioana Danes wrote: > Hello Everyone, > > I have a performance test running with 1200 clients performing this > transaction every second: > > > begin transaction > select nextval('sequence1'); > select nextval('sequence2'); > insert into table1; > insert into table2

[GENERAL] Memory Issue

2011-11-03 Thread Ioana Danes
Hello Everyone, I have a performance test running with 1200 clients performing this transaction every second: begin transaction select nextval('sequence1'); select nextval('sequence2'); insert into table1; insert into table2; commit; Table1 and table2 have no foreign keys and no triggers. Ther

Re: [GENERAL] Memory leak somewhere at PQconnectdb?

2011-09-02 Thread Antonio Vieiro
Hi all, I now know it's somewhat an "academic exercise" of little practical importance, thanks for the clarification!! Cheers, Antonio 2011/9/2 Tom Lane : > Craig Ringer writes: >> Even better, add a valgrind suppressions file for the warnings and >> ignore them. They are "leaks" only in the se

Re: [GENERAL] Memory leak somewhere at PQconnectdb?

2011-09-01 Thread Tom Lane
Craig Ringer writes: > Even better, add a valgrind suppressions file for the warnings and > ignore them. They are "leaks" only in the sense that a static variable > is a leak, ie not at all. Yeah, the bottom line here is that valgrind will warn about many things that are not genuine problems. Yo

Re: [GENERAL] Memory leak somewhere at PQconnectdb?

2011-09-01 Thread Craig Ringer
On 01/09/11 22:08, Antonio Vieiro wrote: > Hi all, > > I'm running one of my programs with valgrind to check for memory leaks > and I'm seeing something like this: You only get the one report, though, right? No matter how many times PQconnectdb is run in a loop? It's internal stuff within OpenSS

Re: [GENERAL] Memory leak somewhere at PQconnectdb?

2011-09-01 Thread Tom Lane
Antonio Vieiro writes: > I'm running one of my programs with valgrind to check for memory leaks > and I'm seeing something like this: > ==13207== 4 bytes in 1 blocks are still reachable in loss record 1 of 256 These are not bugs; they are just permanent allocations that are still there when the

[GENERAL] Memory leak somewhere at PQconnectdb?

2011-09-01 Thread Antonio Vieiro
Hi all, I'm running one of my programs with valgrind to check for memory leaks and I'm seeing something like this: ==13207== 4 bytes in 1 blocks are still reachable in loss record 1 of 256 ==13207==at 0x4026864: malloc (vg_replace_malloc.c:236) ==13207==by 0x43343BD: ??? (in /lib/libcrypt

Re: [GENERAL] Memory management in Postgres

2011-04-13 Thread Jorge Arévalo
2011/4/13 Simon Riggs : > 2011/4/13 Jorge Arévalo : >> >> I'm very interested in PostgreSQL memory management, specially in the >> concept "memory context". I've read the official documentation at >> http://www.postgresql.org/docs/8.4/static/spi-memory.html, but I'd >> like to learn more about it.

Re: [GENERAL] Memory management in Postgres

2011-04-13 Thread Simon Riggs
2011/4/13 Jorge Arévalo : > > I'm very interested in PostgreSQL memory management, specially in the > concept "memory context". I've read the official documentation at > http://www.postgresql.org/docs/8.4/static/spi-memory.html, but I'd > like to learn more about it. Do you recommend me any particu

Re: [GENERAL] Memory leak in SPI_finish call

2011-04-13 Thread Jorge Arévalo
2011/4/5 Jorge Arévalo : > Hello, > > I'm having problems with a PostgreSQL server side C-function. It's not > an aggregate function (operates over a only row of data). When the > function is called over tables with ~4000 rows, it causes postgres > backend crash with SEGFAULT. I know the error is a

[GENERAL] Memory management in Postgres

2011-04-13 Thread Jorge Arévalo
Hello, I'm very interested in PostgreSQL memory management, specially in the concept "memory context". I've read the official documentation at http://www.postgresql.org/docs/8.4/static/spi-memory.html, but I'd like to learn more about it. Do you recommend me any particular book or url? Many thank

[GENERAL] Memory leak in SPI_finish call

2011-04-05 Thread Jorge Arévalo
Hello, I'm having problems with a PostgreSQL server side C-function. It's not an aggregate function (operates over a only row of data). When the function is called over tables with ~4000 rows, it causes postgres backend crash with SEGFAULT. I know the error is a kind of "cumulative", because with

Re: [GENERAL] Memory Errors

2010-09-21 Thread Tom Lane
Sam Nelson writes: > Okay, we're finally getting the last bits of corruption fixed, and I finally > remembered to ask my boss about the kill script. > The only details I have are these: > 1) The script does nothing if there are fewer than 1000 locks on tables in > the database > 2) If there are

Re: [GENERAL] Memory Errors

2010-09-21 Thread Merlin Moncure
On Tue, Sep 21, 2010 at 12:57 PM, Sam Nelson wrote: >> On Thu, Sep 9, 2010 at 8:14 AM, Merlin Moncure wrote: >> Naturally people are going to be skeptical of ec2 since you are so >> abstracted from the hardware. Maybe all your problems stem from a >> single explainable incident -- but we definit

Re: [GENERAL] Memory Errors

2010-09-21 Thread Sam Nelson
Okay, we're finally getting the last bits of corruption fixed, and I finally remembered to ask my boss about the kill script. The only details I have are these: 1) The script does nothing if there are fewer than 1000 locks on tables in the database 2) If there are 1000 or more locks, it will gra

Re: [GENERAL] Memory Errors

2010-09-09 Thread Merlin Moncure
On Wed, Sep 8, 2010 at 6:55 PM, Sam Nelson wrote: > Even if the corruption wasn't a result of that, we weren't too excited about > the process being there to begin with.  We thought there had to be a better > solution than just killing the processes.  So we had a discussion about the > intent of t

Re: [GENERAL] Memory Errors

2010-09-08 Thread Sam Nelson
My (our) complaints about EC2 aren't particularly extensive, but last time I posted to the mailing list saying they were using EC2, the first reply was someone saying that the corruption was the fault of EC2. Not that we don't have complaints at all (there are some aspects that are very frustratin

Re: [GENERAL] Memory Errors

2010-09-08 Thread Tom Lane
Merlin Moncure writes: > On Wed, Sep 8, 2010 at 4:03 PM, Sam Nelson wrote: >> So ... yes, it seems that those four id's are somehow part of the problem. >> They're on amazon EC2 boxes (yeah, we're not too fond of the EC2 boxes >> either), so memtest isn't available, but no new corruption has crop

Re: [GENERAL] Memory Errors

2010-09-08 Thread Merlin Moncure
On Wed, Sep 8, 2010 at 4:03 PM, Sam Nelson wrote: > It figures I'd have an idea right after posting to the mailing list. > Yeah, running COPY foo TO stdout; gets me a list of data before erroring > out, so I did a copy (select * from foo order by id asc) to stdout; to see > if I could make some ki

Re: [GENERAL] Memory Errors

2010-09-08 Thread Sam Nelson
It figures I'd have an idea right after posting to the mailing list. Yeah, running COPY foo TO stdout; gets me a list of data before erroring out, so I did a copy (select * from foo order by id asc) to stdout; to see if I could make some kind of guess as to whether this was related to a single row

Re: [GENERAL] Memory Errors

2010-09-08 Thread Tom Lane
Sam Nelson writes: > pg_dump: Error message from server: ERROR: invalid memory alloc request > size 18446744073709551613 > pg_dump: The command was: COPY public.foo () TO stdout; > That seems like an incredibly large memory allocation request - it shouldn't > be possible for the table to really

Re: [GENERAL] Memory Errors

2010-09-08 Thread Scott Marlowe
On Wed, Sep 8, 2010 at 12:56 PM, Sam Nelson wrote: > Hey, a client of ours has been having some data corruption in their > database.  We got the data corruption fixed and we believe we've discovered > the cause (they had a script killing any waiting queries if the locks on > their database hit 100

[GENERAL] Memory Errors

2010-09-08 Thread Sam Nelson
Hey, a client of ours has been having some data corruption in their database. We got the data corruption fixed and we believe we've discovered the cause (they had a script killing any waiting queries if the locks on their database hit 1000), but they're still getting errors from one table: pg_dum

Re: [GENERAL] Memory Usage and OpenBSD

2010-02-13 Thread Greg Smith
Jeff Ross wrote: Hopefully if I can get it to run well under pgbench the same setup will work well with drupal. The site I was worried about when I went to this bigger server has started a little slower than originally projected so the old server is handling the load. The standard TPC-B-like

Re: [GENERAL] Memory Usage and OpenBSD

2010-02-13 Thread Jeff Ross
Greg Smith wrote: Jeff Ross wrote: I think I'm doing it right. Here's the whole script. I run it from another server on the lan. That looks basically sane--your description was wrong, not your program, which is always better than the other way around. Note that everything your script is d

Re: [GENERAL] Memory Usage and OpenBSD

2010-02-12 Thread Greg Smith
Jeff Ross wrote: I think I'm doing it right. Here's the whole script. I run it from another server on the lan. That looks basically sane--your description was wrong, not your program, which is always better than the other way around. Note that everything your script is doing and way more i

Re: [GENERAL] Memory Usage and OpenBSD

2010-02-11 Thread Jeff Ross
Greg Smith wrote: Jeff Ross wrote: pgbench is run with this: pgbench -h varley.openvistas.net -U _postgresql -t 2 -c $SCALE pgbench with scale starting at 10 and then incrementing by 10. I call it three times for each scale. I've turned on logging to 'all' to try and help figure out whe

Re: [GENERAL] Memory Usage and OpenBSD

2010-02-10 Thread Greg Smith
Jeff Ross wrote: pgbench is run with this: pgbench -h varley.openvistas.net -U _postgresql -t 2 -c $SCALE pgbench with scale starting at 10 and then incrementing by 10. I call it three times for each scale. I've turned on logging to 'all' to try and help figure out where the system panics

Re: [GENERAL] Memory Usage and OpenBSD

2010-02-10 Thread Jeff Ross
Tom Lane wrote: Martijn van Oosterhout writes: On Tue, Feb 09, 2010 at 08:19:51PM +0500, Anton Maksimenkov wrote: Can anybody briefly explain me how one postgres process allocate memory for it needs? There's no real maximum, as it depends on the exact usage. However, in ge

Re: [GENERAL] Memory Usage and OpenBSD

2010-02-10 Thread Tom Lane
Martijn van Oosterhout writes: > On Tue, Feb 09, 2010 at 08:19:51PM +0500, Anton Maksimenkov wrote: >> Can anybody briefly explain me how one postgres process allocate >> memory for it needs? > There's no real maximum, as it depends on the exact usage. However, in > general postgres tries to keep

Re: [GENERAL] Memory Usage and OpenBSD

2010-02-10 Thread Anton Maksimenkov
2010/2/10 Martijn van Oosterhout : >> Can anybody briefly explain me how one postgres process allocate >> memory for it needs? > > There's no real maximum, as it depends on the exact usage. However, in > general postgres tries to keep below the values in work_mem and > maintainence_workmem. Most of

Re: [GENERAL] Memory Usage and OpenBSD

2010-02-09 Thread Martijn van Oosterhout
On Tue, Feb 09, 2010 at 08:19:51PM +0500, Anton Maksimenkov wrote: > It means that on openbsd i386 we have about 2,2G of virtual space for > malloc, shm*. So, postgres will use that space. > > But mmap() use random addresses. So when you get big chunk of memory > for shared buffers (say, 2G) then

Re: [GENERAL] Memory Usage and OpenBSD

2010-02-09 Thread Anton Maksimenkov
2010/2/9 Scott Marlowe : > On Tue, Feb 9, 2010 at 3:18 AM, Anton Maksimenkov wrote: >>> Isn't the usual advice here is to log the ulimit setting from the pg >>> startup script so you can what it really is for the user at the moment >> I think that "su" is enough: > In previous discussions it was m

Re: [GENERAL] Memory Usage and OpenBSD

2010-02-09 Thread Scott Marlowe
On Tue, Feb 9, 2010 at 3:18 AM, Anton Maksimenkov wrote: > 2010/1/28 Scott Marlowe : >>> related to maximum per-process data space.  I don't know BSD very well >>> so I can't say if datasize is the only such value for BSD, but it'd be >>> worth checking.  (Hmm, on OS X which is at least partly BSD

Re: [GENERAL] Memory Usage and OpenBSD

2010-02-09 Thread Anton Maksimenkov
2010/1/28 Scott Marlowe : >> related to maximum per-process data space.  I don't know BSD very well >> so I can't say if datasize is the only such value for BSD, but it'd be >> worth checking.  (Hmm, on OS X which is at least partly BSDish, I see >> -m and -v in addition to -d, so I'm suspicious Op

Re: [GENERAL] Memory Usage and OpenBSD

2010-01-27 Thread Scott Marlowe
On Wed, Jan 27, 2010 at 4:42 PM, Tom Lane wrote: > related to maximum per-process data space.  I don't know BSD very well > so I can't say if datasize is the only such value for BSD, but it'd be > worth checking.  (Hmm, on OS X which is at least partly BSDish, I see > -m and -v in addition to -d,

Re: [GENERAL] Memory Usage and OpenBSD

2010-01-27 Thread Tom Lane
Jeff Ross writes: > Tom Lane wrote: >> Better look at the "ulimit" values the postmaster is started with; > OpenBSD makes a _postgresql user on install and it is in the daemon class > with > the following values: > daemon:\ > :ignorenologin:\ > :datasize=infinity:\ >

Re: [GENERAL] Memory Usage and OpenBSD

2010-01-27 Thread Jeff Ross
Tom Lane wrote: Jeff Ross writes: Running a simple select only pgbench test against it will fail with an out of memory error as it tries to vacuum --analyze the newly created database with 750 tuples. Better look at the "ulimit" values the postmaster is started with; you shouldn't be get

Re: [GENERAL] Memory Usage and OpenBSD

2010-01-27 Thread Tom Lane
Jeff Ross writes: > Running a simple select only pgbench test against it will fail with an out of > memory error as it tries to vacuum --analyze the newly created database with > 750 tuples. Better look at the "ulimit" values the postmaster is started with; you shouldn't be getting that out-

[GENERAL] Memory Usage and OpenBSD

2010-01-27 Thread Jeff Ross
I'm not getting something about the best way to set up a server using PostgreSQL as a backend for a busy web server running drupal. The postgresql performance folks http://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server say that in a server with more that 1GB of ram "a reasonable sta

Re: [GENERAL] Memory Access Violation While Using PQexec

2010-01-14 Thread Craig Ringer
On 14/01/2010 4:49 PM, Yan Cheng Cheok wrote: I encounter case when I call a stored procedure for 299166 th times (intensive, i put a non-stop while true loop to call stored procedure) , the following exception will be thrown from PQexec. I am rather sure the exception are from PQexec, as ther

[GENERAL] Memory Access Violation While Using PQexec

2010-01-14 Thread Yan Cheng Cheok
I encounter case when I call a stored procedure for 299166 th times (intensive, i put a non-stop while true loop to call stored procedure) , the following exception will be thrown from PQexec. I am rather sure the exception are from PQexec, as there is a just before cout and just after cout wra

Re: [GENERAL] memory usage of group by select

2009-12-29 Thread Anthony
On Tue, Dec 29, 2009 at 4:09 PM, Alvaro Herrera wrote: > It's expecting 85k distinct groups. If that's not accurate, then > HashAggregate would use more memory than expected. Great diagnosis. There are actually about 76 million distinct groups. > See if you can make it work by setting enable

Re: [GENERAL] memory usage of group by select

2009-12-29 Thread Tom Lane
Alvaro Herrera writes: > It's expecting 85k distinct groups. If that's not accurate, then > HashAggregate would use more memory than expected. See if you can make > it work by setting enable_hashagg = off. > If that works, good -- the real solution is different. Maybe you need > to ANALYZE mor

Re: [GENERAL] memory usage of group by select

2009-12-29 Thread Alvaro Herrera
Anthony wrote: > On Tue, Dec 29, 2009 at 3:41 PM, Anthony wrote: > > > I'm running a group by query on a table with over a billion rows and my > > memory usage is seemingly growing without bounds. Eventually the mem usage > > exceeds my physical memory and everything starts swapping. > > > > I

Re: [GENERAL] memory usage of group by select

2009-12-29 Thread Anthony
On Tue, Dec 29, 2009 at 3:41 PM, Anthony wrote: > I'm running a group by query on a table with over a billion rows and my > memory usage is seemingly growing without bounds. Eventually the mem usage > exceeds my physical memory and everything starts swapping. > I guess I didn't ask my question.

[GENERAL] memory usage of group by select

2009-12-29 Thread Anthony
Hi all, I'm running a group by query on a table with over a billion rows and my memory usage is seemingly growing without bounds. Eventually the mem usage exceeds my physical memory and everything starts swapping. Here is what I gather to be the relevant info: My machine has 768 megs of ram. s

Re: ***UNCHECKED*** Re: [GENERAL] memory leak occur when disconnect database

2009-07-21 Thread Tom Lane
Craig Ringer writes: > On Tue, 2009-07-21 at 10:13 -0400, Tom Lane wrote: >> It's not unusual for "top" to show the postmaster's child processes as >> "postmaster" as well. Depends on the platform and the options given >> to top. > Ah. Thanks for clearing that one up. That'd make more sense, sin

Re: ***UNCHECKED*** Re: [GENERAL] memory leak occur when disconnect database

2009-07-21 Thread Craig Ringer
On Tue, 2009-07-21 at 10:13 -0400, Tom Lane wrote: > Craig Ringer writes: > > I'm a bit puzzled about why you have three "postmaster" instances shown > > as running. > > It's not unusual for "top" to show the postmaster's child processes as > "postmaster" as well. Depends on the platform and the

Re: ***UNCHECKED*** Re: [GENERAL] memory leak occur when disconnect database

2009-07-21 Thread Tom Lane
Craig Ringer writes: > I'm a bit puzzled about why you have three "postmaster" instances shown > as running. It's not unusual for "top" to show the postmaster's child processes as "postmaster" as well. Depends on the platform and the options given to top. regards, tom la

Re: ***UNCHECKED*** Re: [GENERAL] memory leak occur when disconnect database

2009-07-21 Thread Craig Ringer
On Tue, 2009-07-21 at 19:39 +0900, tanjunhua wrote: > I'm sorry for the twice action. because the mail server reject my response. > I should compress it with ciper code(11) and the execute program is > compressed also. When I build your example from source I see no indication of anything wro

Re: [GENERAL] memory leak occur when disconnect database

2009-07-20 Thread Craig Ringer
On Tue, 2009-07-21 at 13:53 +0900, tanjunhua wrote: > I get the memory leak scenario not only from Valgrind, but also from the > output of top command. > At first I think the memory leak occur when I disconnect database by > Valgrind, then I write a test sample that just connect and disconnect

Re: [GENERAL] memory leak occur when disconnect database

2009-07-20 Thread tanjunhua
Because of the three-day break, my response is late. 8.1.8 is pretty old. Also you'll have better luck getting help if you actually include the output from Valgrind. the output from Valgrind is not stored. from now on, I will do it again and get the result from Valgrind. PS: the memory leak

Re: [GENERAL] memory leak occur when disconnect database

2009-07-20 Thread tanjunhua
Because of the three-day break, my response is late. Valgrind is a great tool, but you must learn how to identify false positives and tell the difference between a leak that matters (say 1kb allocated and not freed in a loop that runs once per second) and a leak that doesn't. I get the memory

Re: [GENERAL] memory leak occur when disconnect database

2009-07-17 Thread Craig Ringer
Sorry for the reply-to-self, but I thought I'd take ecpg out of the equation: #include #include int main() { struct passwd p; struct passwd * r; char buf[500]; getpwuid_r(1000, &p, &buf[0], 500, &r); } ... produces the same leak report. Since you didn't include information li

Re: [GENERAL] memory leak occur when disconnect database

2009-07-17 Thread Craig Ringer
Your test case doesn't build, but I've attached a trivially tweaked one that does. Valgrind's report (valgrind --leak-check=full ./test) on my Ubuntu 9.04 machine with Pg 8.3.7 is: ==23382== 156 (36 direct, 120 indirect) bytes in 1 blocks are definitely lost in loss record 1 of 4 ==23382==at

Re: [GENERAL] memory leak occur when disconnect database

2009-07-17 Thread Walton Hoops
tgres General Postgres General Subject: [GENERAL] memory leak occur when disconnect database I'm running postgres 8.1.8 on Debian and I think memory leak occur when disconnect database. 1. environment setting 1.1 postgresql version:

[GENERAL] memory leak occur when disconnect database

2009-07-17 Thread tanjunhua
I'm running postgres 8.1.8 on Debian and I think memory leak occur when disconnect database. 1. environment setting 1.1 postgresql version: version -

Re: [GENERAL] Memory use in 8.3 plpgsql with heavy use of xpath()

2008-08-10 Thread Matt Magoffin
> These numbers don't even have any demonstrable connection to Postgres, > let alone to an xpath-related memory leak. You're going to need to come > up with a concrete test case if you want anyone to investigate. > > regards, tom lane As I said in the start of this thread, t

<    1   2   3   4   >