Re: [HACKERS] New statistics for WAL buffer dirty writes

2013-09-17 Thread Fujii Masao
On Wed, Sep 11, 2013 at 12:43 PM, Satoshi Nagayasu sn...@uptime.jp wrote: (2013/09/10 22:48), Peter Eisentraut wrote: On 9/10/13 3:37 AM, Satoshi Nagayasu wrote: Thanks for checking. Revised one attached. Please fix compiler warning: walwriter.c: In function ‘WalWriterMain’:

Re: [HACKERS] New statistics for WAL buffer dirty writes

2013-09-10 Thread Satoshi Nagayasu
Thanks for checking. Revised one attached. (2013/09/10 6:43), Peter Eisentraut wrote: On 9/6/13 11:32 PM, Satoshi Nagayasu wrote: The revised patch for wal buffer statistics is attached. A test script is also attached. Please take a look. You have duplicate OIDs. Run the script

Re: [HACKERS] New statistics for WAL buffer dirty writes

2013-09-10 Thread Peter Eisentraut
On 9/10/13 3:37 AM, Satoshi Nagayasu wrote: Thanks for checking. Revised one attached. Please fix compiler warning: walwriter.c: In function ‘WalWriterMain’: walwriter.c:293:3: warning: implicit declaration of function ‘pgstat_send_walwriter’ [-Wimplicit-function-declaration] -- Sent via

Re: [HACKERS] New statistics for WAL buffer dirty writes

2013-09-10 Thread Satoshi Nagayasu
(2013/09/10 22:48), Peter Eisentraut wrote: On 9/10/13 3:37 AM, Satoshi Nagayasu wrote: Thanks for checking. Revised one attached. Please fix compiler warning: walwriter.c: In function ‘WalWriterMain’: walwriter.c:293:3: warning: implicit declaration of function ‘pgstat_send_walwriter’

Re: [HACKERS] New statistics for WAL buffer dirty writes

2013-09-09 Thread Peter Eisentraut
On 9/6/13 11:32 PM, Satoshi Nagayasu wrote: The revised patch for wal buffer statistics is attached. A test script is also attached. Please take a look. You have duplicate OIDs. Run the script duplicate_oids to find them. -- Sent via pgsql-hackers mailing list

Re: [HACKERS] New statistics for WAL buffer dirty writes

2013-09-09 Thread Peter Geoghegan
On Mon, Sep 9, 2013 at 2:43 PM, Peter Eisentraut pete...@gmx.net wrote: You have duplicate OIDs. Run the script duplicate_oids to find them. Are you considering picking up the script that Andrew wrote to automate that as part of the build? I wonder why that didn't end up going anywhere. --

Re: [HACKERS] New statistics for WAL buffer dirty writes

2013-09-09 Thread Peter Eisentraut
On Mon, 2013-09-09 at 14:51 -0700, Peter Geoghegan wrote: On Mon, Sep 9, 2013 at 2:43 PM, Peter Eisentraut pete...@gmx.net wrote: You have duplicate OIDs. Run the script duplicate_oids to find them. Are you considering picking up the script that Andrew wrote to automate that as part of the

Re: [HACKERS] New statistics for WAL buffer dirty writes

2013-09-09 Thread Peter Geoghegan
On Mon, Sep 9, 2013 at 6:05 PM, Peter Eisentraut pete...@gmx.net wrote: It is automated. Oh, yeah. I see that the maintainer-check target does that. I should probably get into the habit of using targets other than check/installcheck, as you recently demonstrated. -- Peter Geoghegan -- Sent

Re: [HACKERS] New statistics for WAL buffer dirty writes

2013-09-06 Thread Satoshi Nagayasu
Hi, The revised patch for wal buffer statistics is attached. A test script is also attached. Please take a look. Regards, (2013/07/19 7:49), Satoshi Nagayasu wrote: Will revise and re-resubmit for the next CF. Regards, 2013/07/19 1:06, Alvaro Herrera wrote: What happened to this patch?

Re: [HACKERS] New statistics for WAL buffer dirty writes

2013-07-18 Thread Alvaro Herrera
What happened to this patch? We were waiting on an updated version from you. Satoshi Nagayasu wrote: (2012/12/10 3:06), Tomas Vondra wrote: On 29.10.2012 04:58, Satoshi Nagayasu wrote: 2012/10/24 1:12, Alvaro Herrera wrote: Satoshi Nagayasu escribi�: With this patch, walwriter process

Re: [HACKERS] New statistics for WAL buffer dirty writes

2013-07-18 Thread Satoshi Nagayasu
Will revise and re-resubmit for the next CF. Regards, 2013/07/19 1:06, Alvaro Herrera wrote: What happened to this patch? We were waiting on an updated version from you. Satoshi Nagayasu wrote: (2012/12/10 3:06), Tomas Vondra wrote: On 29.10.2012 04:58, Satoshi Nagayasu wrote:

Re: [HACKERS] New statistics for WAL buffer dirty writes

2013-01-20 Thread Satoshi Nagayasu
(2012/11/27 7:42), Alvaro Herrera wrote: Satoshi Nagayasu escribió: I attached the latest one, which splits the reset_time for bgwriter and walwriter, and provides new system view, called pg_stat_walwriter, to show the dirty write counter and the reset time. Thanks. I gave this a look and I

Re: [HACKERS] New statistics for WAL buffer dirty writes

2013-01-20 Thread Satoshi Nagayasu
(2012/12/10 3:06), Tomas Vondra wrote: On 29.10.2012 04:58, Satoshi Nagayasu wrote: 2012/10/24 1:12, Alvaro Herrera wrote: Satoshi Nagayasu escribi�: With this patch, walwriter process and each backend process would sum up dirty writes, and send it to the stat collector. So, the value could

Re: [HACKERS] New statistics for WAL buffer dirty writes

2012-12-09 Thread Tomas Vondra
On 29.10.2012 04:58, Satoshi Nagayasu wrote: 2012/10/24 1:12, Alvaro Herrera wrote: Satoshi Nagayasu escribi�: With this patch, walwriter process and each backend process would sum up dirty writes, and send it to the stat collector. So, the value could be saved in the stat file, and could be

Re: [HACKERS] New statistics for WAL buffer dirty writes

2012-11-26 Thread Alvaro Herrera
Satoshi Nagayasu escribió: I attached the latest one, which splits the reset_time for bgwriter and walwriter, and provides new system view, called pg_stat_walwriter, to show the dirty write counter and the reset time. Thanks. I gave this a look and I have a couple of comments: 1. The

Re: [HACKERS] New statistics for WAL buffer dirty writes

2012-10-28 Thread Satoshi Nagayasu
2012/10/24 1:12, Alvaro Herrera wrote: Satoshi Nagayasu escribió: With this patch, walwriter process and each backend process would sum up dirty writes, and send it to the stat collector. So, the value could be saved in the stat file, and could be kept on restarting. The statistics could be

Re: [HACKERS] New statistics for WAL buffer dirty writes

2012-10-23 Thread Alvaro Herrera
Satoshi Nagayasu escribió: With this patch, walwriter process and each backend process would sum up dirty writes, and send it to the stat collector. So, the value could be saved in the stat file, and could be kept on restarting. The statistics could be retreive with using

Re: [HACKERS] New statistics for WAL buffer dirty writes

2012-09-25 Thread Satoshi Nagayasu
Hi, 2012/08/12 7:11, Jeff Janes wrote: On Sat, Jul 28, 2012 at 3:33 PM, Jeff Janes jeff.ja...@gmail.com wrote: On Sat, Jul 7, 2012 at 9:17 PM, Satoshi Nagayasu sn...@uptime.jp wrote: Hi, Jeff Janes has pointed out that my previous patch could hold a number of the dirty writes only in single

Re: [HACKERS] New statistics for WAL buffer dirty writes

2012-08-21 Thread Robert Haas
On Sat, Aug 11, 2012 at 6:11 PM, Jeff Janes jeff.ja...@gmail.com wrote: However, I do think we will want to add a way to query for the time of the last reset, as other monitoring features are going that way. That should be easy to add. Is it OK that the count is reset upon a server restart?

Re: [HACKERS] New statistics for WAL buffer dirty writes

2012-08-11 Thread Jeff Janes
On Sat, Jul 28, 2012 at 3:33 PM, Jeff Janes jeff.ja...@gmail.com wrote: On Sat, Jul 7, 2012 at 9:17 PM, Satoshi Nagayasu sn...@uptime.jp wrote: Hi, Jeff Janes has pointed out that my previous patch could hold a number of the dirty writes only in single local backend, and it could not hold

Re: [HACKERS] New statistics for WAL buffer dirty writes

2012-08-01 Thread Robert Haas
On Tue, Jul 31, 2012 at 4:06 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: IMHO, the way we have it now is kind of a mess. SpinLockAcquire and SpinLockRelease are required to be CPU barriers, but they are not required to be compiler barriers. If we changed

Re: [HACKERS] New statistics for WAL buffer dirty writes

2012-08-01 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Tue, Jul 31, 2012 at 4:06 PM, Tom Lane t...@sss.pgh.pa.us wrote: I agree that volatile-izing everything in the vicinity is a sucky solution, but the last time we looked at this there did not seem to be a better one. Well, Linux has a barrier()

Re: [HACKERS] New statistics for WAL buffer dirty writes

2012-08-01 Thread Robert Haas
On Wed, Aug 1, 2012 at 10:12 AM, Tom Lane t...@sss.pgh.pa.us wrote: Solving the problem for linux only, or gcc only, isn't going to get us to a place where we can stop volatile-izing call sites. We need to be sure it works for every single case supported by s_lock.h. Yep, that's the problem

Re: [HACKERS] New statistics for WAL buffer dirty writes

2012-07-31 Thread Robert Haas
On Sat, Jul 28, 2012 at 6:33 PM, Jeff Janes jeff.ja...@gmail.com wrote: A concern I have is whether the XLogCtlWrite *Write pointer needs to be declared volatile, to prevent the compiler from pushing operations on them outside of the locks (and so memory barriers) that formally protect them.

Re: [HACKERS] New statistics for WAL buffer dirty writes

2012-07-31 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: IMHO, the way we have it now is kind of a mess. SpinLockAcquire and SpinLockRelease are required to be CPU barriers, but they are not required to be compiler barriers. If we changed that so that they were required to act as barriers of both flavors,

Re: [HACKERS] New statistics for WAL buffer dirty writes

2012-07-29 Thread Simon Riggs
On 7 July 2012 18:06, Robert Haas robertmh...@gmail.com wrote: Sure, but I doubt that would be as informative as this. It's no big deal if you hit 100% every once in a while; what you really want to know is whether it's happening once per second or once per week. Agreed. I can't see an

Re: [HACKERS] New statistics for WAL buffer dirty writes

2012-07-28 Thread Jeff Janes
On Sat, Jul 7, 2012 at 9:17 PM, Satoshi Nagayasu sn...@uptime.jp wrote: Hi, Jeff Janes has pointed out that my previous patch could hold a number of the dirty writes only in single local backend, and it could not hold all over the cluster, because the counter was allocated in the local

[HACKERS] New statistics for WAL buffer dirty writes

2012-07-07 Thread Satoshi Nagayasu
Hi all, I've created new patch to get/reset statistics of WAL buffer writes (flushes) caused by WAL buffer full. This patch provides two new functions, pg_stat_get_xlog_dirty_write() and pg_stat_reset_xlog_dirty_write(), which have been designed to determine an appropriate value for WAL buffer

Re: [HACKERS] New statistics for WAL buffer dirty writes

2012-07-07 Thread Euler Taveira
On 07-07-2012 09:00, Satoshi Nagayasu wrote: I've created new patch to get/reset statistics of WAL buffer writes (flushes) caused by WAL buffer full. This new statistic doesn't solve your problem (tune wal_buffers). It doesn't give you the wal_buffers value. It only says hey, I needed more

Re: [HACKERS] New statistics for WAL buffer dirty writes

2012-07-07 Thread Satoshi Nagayasu
2012/07/07 22:07, Euler Taveira wrote: On 07-07-2012 09:00, Satoshi Nagayasu wrote: I've created new patch to get/reset statistics of WAL buffer writes (flushes) caused by WAL buffer full. This new statistic doesn't solve your problem (tune wal_buffers). It doesn't give you the wal_buffers

Re: [HACKERS] New statistics for WAL buffer dirty writes

2012-07-07 Thread Robert Haas
On Jul 7, 2012, at 9:07 AM, Euler Taveira eu...@timbira.com wrote: On 07-07-2012 09:00, Satoshi Nagayasu wrote: I've created new patch to get/reset statistics of WAL buffer writes (flushes) caused by WAL buffer full. This new statistic doesn't solve your problem (tune wal_buffers). It

Re: [HACKERS] New statistics for WAL buffer dirty writes

2012-07-07 Thread Magnus Hagander
On Sat, Jul 7, 2012 at 3:52 PM, Robert Haas robertmh...@gmail.com wrote: On Jul 7, 2012, at 9:07 AM, Euler Taveira eu...@timbira.com wrote: On 07-07-2012 09:00, Satoshi Nagayasu wrote: I've created new patch to get/reset statistics of WAL buffer writes (flushes) caused by WAL buffer full.

Re: [HACKERS] New statistics for WAL buffer dirty writes

2012-07-07 Thread Robert Haas
On Jul 7, 2012, at 8:54 AM, Magnus Hagander mag...@hagander.net wrote: On Sat, Jul 7, 2012 at 3:52 PM, Robert Haas robertmh...@gmail.com wrote: On Jul 7, 2012, at 9:07 AM, Euler Taveira eu...@timbira.com wrote: On 07-07-2012 09:00, Satoshi Nagayasu wrote: I've created new patch to get/reset

Re: [HACKERS] New statistics for WAL buffer dirty writes

2012-07-07 Thread Magnus Hagander
On Sat, Jul 7, 2012 at 7:06 PM, Robert Haas robertmh...@gmail.com wrote: On Jul 7, 2012, at 8:54 AM, Magnus Hagander mag...@hagander.net wrote: On Sat, Jul 7, 2012 at 3:52 PM, Robert Haas robertmh...@gmail.com wrote: On Jul 7, 2012, at 9:07 AM, Euler Taveira eu...@timbira.com wrote: On

Re: [HACKERS] New statistics for WAL buffer dirty writes

2012-07-07 Thread Satoshi Nagayasu
Hi, Jeff Janes has pointed out that my previous patch could hold a number of the dirty writes only in single local backend, and it could not hold all over the cluster, because the counter was allocated in the local process memory. That's true, and I have fixed it with moving the counter into the