Re: [GENERAL] Increase in max_connections

2014-04-08 Thread Kevin Grittner
Kevin Grittner kgri...@ymail.com wrote: Please post the output of this: numactl --hardware Oh, it would also help in making specific suggestions if you could show the output of: mount | grep cpuset ... and a listing of file names in the mounted directory. There is some variation among

Re: [GENERAL] Increase in max_connections

2014-04-07 Thread Kevin Grittner
Anand Kumar, Karthik karthik.anandku...@classmates.com wrote: We finally made some headway on this - we noticed messages like the below in /var/log/messages whenever the issue happened: Mar 26 07:39:58 site-db01b kernel: postmaster: page allocation failure. Anyone have any idea why memory

Re: [GENERAL] Increase in max_connections

2014-03-26 Thread Anand Kumar, Karthik
Hi all, We finally made some headway on this - we noticed messages like the below in /var/log/messages whenever the issue happened: Mar 26 07:39:58 site-db01b kernel: postmaster: page allocation failure. order:1, mode:0x20 Mar 26 07:39:58 site-db01b kernel: Pid: 39066, comm: postmaster Not

Re: [GENERAL] Increase in max_connections

2014-03-26 Thread Bruce Momjian
On Wed, Mar 26, 2014 at 08:22:01PM +, Anand Kumar, Karthik wrote: Looking a little deeper, I saw signs of memory being heavily fragmented: root@site-db01b:/var/log # cat /proc/buddyinfo Node 0, zone DMA 1 1 2 2 2 1 0 0 1 1 3 Node 0, zone DMA32 8 7 8 7 10 8 7 11 9 5 92 Node 0, zone

Re: [GENERAL] Increase in max_connections

2014-03-26 Thread Anand Kumar, Karthik
Thanks Bruce. Really interesting, but, I show zone reclaim is already turned off on our system. root@site-db01b:~ # numactl --hardware available: 2 nodes (0-1) node 0 cpus: 0 1 2 3 4 5 12 13 14 15 16 17 node 0 size: 393181 MB node 0 free: 467 MB node 1 cpus: 6 7 8 9 10 11 18 19 20 21 22 23 node 1

Re: [GENERAL] Increase in max_connections

2014-03-14 Thread Anand Kumar, Karthik
For anyone that's still following - we tried upgrading to postgres 9.3.3 - that hasn't helped. Running an strace on the pid that was consuming the highest CPU at the time of the outage shows: semop(91521110, {{12, -1, 0}}, 1) = 0 semop(91521110, {{12, -1, 0}}, 1) = 0 semop(91521110,

Re: [GENERAL] Increase in max_connections

2014-03-11 Thread Igor Neyman
From: pgsql-general-ow...@postgresql.org [mailto:pgsql-general-ow...@postgresql.org] On Behalf Of Anand Kumar, Karthik Sent: Monday, March 10, 2014 9:04 PM To: pgsql-general@postgresql.org Subject: [GENERAL] Increase in max_connections Hi all, We're running postgres 9.3.2, server configuration

Re: [GENERAL] Increase in max_connections

2014-03-11 Thread Jeff Janes
On Mon, Mar 10, 2014 at 6:04 PM, Anand Kumar, Karthik karthik.anandku...@classmates.com wrote: Hi all, We're running postgres 9.3.2, server configuration below. Seemingly randomly, we will see the number of active queries in postgres go up until we hit max_connections. The DB will

Re: [GENERAL] Increase in max_connections

2014-03-11 Thread Anand Kumar, Karthik
To: Anand Kumar, Karthik karthik.anandku...@classmates.commailto:karthik.anandku...@classmates.com Cc: pgsql-general@postgresql.orgmailto:pgsql-general@postgresql.org pgsql-general@postgresql.orgmailto:pgsql-general@postgresql.org Subject: Re: [GENERAL] Increase in max_connections On Tue, Mar 11, 2014

Re: [GENERAL] Increase in max_connections

2014-03-11 Thread Anand Kumar, Karthik
To: Anand Kumar, Karthik karthik.anandku...@classmates.commailto:karthik.anandku...@classmates.com Cc: pgsql-general@postgresql.orgmailto:pgsql-general@postgresql.org pgsql-general@postgresql.orgmailto:pgsql-general@postgresql.org Subject: Re: [GENERAL] Increase in max_connections On Mon, Mar 10

Re: [GENERAL] Increase in max_connections

2014-03-11 Thread Jeff Janes
On Tue, Mar 11, 2014 at 10:20 AM, Anand Kumar, Karthik karthik.anandku...@classmates.com wrote: Thanks Jeff. We have scripts in place now to capture the incoming rate of requests. Waiting on the crash to happen to see if it spikes up :) Re: min_log_duration - we *do* see a good number of

Re: [GENERAL] Increase in max_connections

2014-03-11 Thread John R Pierce
On 3/11/2014 10:20 AM, Anand Kumar, Karthik wrote: We typically see about 500-700 active queries at a time if these are primarily small/fast queries, like OLTP operations, and you DONT have 200-400 CPU cores on this server, you will likely find that if you use a queueing mechanism to only

[GENERAL] Increase in max_connections

2014-03-10 Thread Anand Kumar, Karthik
Hi all, We're running postgres 9.3.2, server configuration below. Seemingly randomly, we will see the number of active queries in postgres go up until we hit max_connections. The DB will recover after a few minutes. We had the issue a couple of times in Feb 2014. We then upgraded the postgres

Re: [GENERAL] Increase in max_connections

2014-03-10 Thread Venkata Balaji Nagothi
On Tue, Mar 11, 2014 at 12:04 PM, Anand Kumar, Karthik karthik.anandku...@classmates.com wrote: Hi all, We're running postgres 9.3.2, server configuration below. Seemingly randomly, we will see the number of active queries in postgres go up until we hit max_connections. The DB will