Re: [HACKERS] gaussian distribution pgbench

2014-03-18 Thread KONDO Mitsumasa
And I find new useful point of this feature. Under following results are '--gaussian=20' case and '--gaussian=2' case, and postgresql setting is same. > [mitsu-ko@pg-rex31 pgbench]$ ./pgbench -c8 -j4 --gaussian=20 -T30 -P 5 > starting vacuum...end. > progress: 5.0 s, 4285.8 tps, lat 1.860 ms stdde

Re: [HACKERS] gaussian distribution pgbench

2014-03-18 Thread KONDO Mitsumasa
(2014/03/17 23:29), Robert Haas wrote: On Sat, Mar 15, 2014 at 4:50 AM, Mitsumasa KONDO wrote: There are explanations and computations as comments in the code. If it is about the documentation, I'm not sure that a very precise mathematical definition will help a lot of people, and might rather

Re: [HACKERS] gaussian distribution pgbench

2014-03-18 Thread KONDO Mitsumasa
(2014/03/17 22:37), Tom Lane wrote: > KONDO Mitsumasa writes: >> (2014/03/17 18:02), Heikki Linnakangas wrote: >>> On 03/17/2014 10:40 AM, KONDO Mitsumasa wrote: >>> There is an infinite number of variants of the TPC-B test that we could >>> include >>

Re: [HACKERS] gaussian distribution pgbench

2014-03-17 Thread KONDO Mitsumasa
(2014/03/17 18:02), Heikki Linnakangas wrote: On 03/17/2014 10:40 AM, KONDO Mitsumasa wrote: By the way, you seem to want to remove --gaussian=NUM and --exponential=NUM command options. Can you tell me the objective reason? I think pgbench is the benchmark test on PostgreSQL and default

Re: [HACKERS] gaussian distribution pgbench

2014-03-17 Thread KONDO Mitsumasa
(2014/03/17 17:46), Heikki Linnakangas wrote: On 03/15/2014 08:53 AM, Fabien COELHO wrote: >* Does min and max really make sense for gaussian and exponential >distributions? For gaussian, I would expect mean and standard deviation as >the parameters, not min/max/threshold. Yes... and no:-) The

Re: [HACKERS] gaussian distribution pgbench

2014-03-17 Thread KONDO Mitsumasa
Hi Heikki-san, (2014/03/17 14:39), KONDO Mitsumasa wrote: (2014/03/15 15:53), Fabien COELHO wrote: Hello Heikki, A couple of comments: * There should be an explicit "\setrandom ... uniform" option too, even though you get that implicitly if you don't specify the distributi

Re: [HACKERS] gaussian distribution pgbench

2014-03-16 Thread KONDO Mitsumasa
(2014/03/15 15:53), Fabien COELHO wrote: Hello Heikki, A couple of comments: * There should be an explicit "\setrandom ... uniform" option too, even though you get that implicitly if you don't specify the distribution Indeed. I agree. I suggested it, but it got lost. * What exactly does t

Re: [HACKERS] gaussian distribution pgbench

2014-03-13 Thread KONDO Mitsumasa
(2014/03/13 23:00), Fujii Masao wrote: On Thu, Mar 13, 2014 at 10:51 PM, Heikki Linnakangas wrote: On 03/13/2014 03:17 PM, Fujii Masao wrote: On Tue, Mar 11, 2014 at 1:49 PM, KONDO Mitsumasa wrote: (2014/03/09 1:49), Fabien COELHO wrote: I'm okay with this UI and itsaccess probab

Re: [HACKERS] gaussian distribution pgbench

2014-03-13 Thread KONDO Mitsumasa
Hi, (2014/03/14 4:21), Fabien COELHO wrote: We should do the same discussion for the UI of command-line option? The patch adds two options --gaussian and --exponential, but this UI seems to be a bit inconsistent with the UI for \setrandom. Instead, we can use something like --distribution=[uni

Re: [HACKERS] gaussian distribution pgbench

2014-03-10 Thread KONDO Mitsumasa
(2014/03/09 1:49), Fabien COELHO wrote: Hello Mitsumasa-san, New "\setrandom" interface is here. \setrandom var min max [gaussian threshold | exponential threshold] Attached patch realizes this interface, but it has little bit ugly codeing in executeStatement() and process_commands().. I

Re: [HACKERS] gaussian distribution pgbench

2014-03-06 Thread KONDO Mitsumasa
(2014/03/07 16:02), KONDO Mitsumasa wrote: And other cases are classified under following. \setrandom var min max gaussian #hoge --> uniform Oh, it's wrong... It will be.. \setrandom var min max gaussian #hoge --> ERROR Regards, -- Mitsumasa KONDO NTT Open Source Software Cente

Re: [HACKERS] gaussian distribution pgbench

2014-03-06 Thread KONDO Mitsumasa
Hi, (2014/03/04 17:42), KONDO Mitsumasa wrote:> (2014/03/04 17:28), Fabien COELHO wrote: >>> OK. I'm not sure which idia is the best. So I wait for comments in community:) >> Hmmm. Maybe you can do what Tom voted for, he is the committer:-) > Yeah, but he mi

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2014-03-04 Thread KONDO Mitsumasa
Hi all, I think this patch is completely forgotten, and feel very unfortunate:( Min, max, and stdev is basic statistics in general monitoring tools, So I'd like to push it. (2014/02/12 15:45), KONDO Mitsumasa wrote: (2014/01/29 17:31), Rajeev rastogi wrote: No Issue, you can share m

Re: [HACKERS] gaussian distribution pgbench

2014-03-04 Thread KONDO Mitsumasa
(2014/03/04 17:28), Fabien COELHO wrote: OK. I'm not sure which idia is the best. So I wait for comments in community:) Hmmm. Maybe you can do what Tom voted for, he is the committer:-) Yeah, but he might change his mind by our disscuttion. So I wait untill tomorrow, and if nothing to comment,

Re: [HACKERS] gaussian distribution pgbench

2014-03-03 Thread KONDO Mitsumasa
(2014/03/03 16:51), Fabien COELHO wrote:>>>\setrandom foo 1 10 [uniform] >>>\setrandom foo 1 :size gaussian 3.6 >>>\setrandom foo 1 100 exponential 7.2 >> It's good design. I think it will become more low overhead at part of parsing >> in pgbench, because comparison of strings will be

Re: [HACKERS] gaussian distribution pgbench

2014-03-02 Thread KONDO Mitsumasa
(2014/03/02 22:32), Fabien COELHO wrote: Alvaro Herrera writes: Seems that in the review so far, Fabien has focused mainly in the mathematical properties of the new random number generation. That seems perfectly fine, but no comment has been made about the chosen UI for the feature. Per the f

Re: [HACKERS] What behavior is in this loop?

2014-02-28 Thread KONDO Mitsumasa
(2014/02/27 20:19), Heikki Linnakangas wrote: On 02/27/2014 12:38 PM, KONDO Mitsumasa wrote: I found interesting "for" and "while" loop in WaitForWALToBecomeAvailable() in xlog.c. Can you tell me this behavior? for (;;) { ~ } while (StanbyMode) I confirmed this code

Re: [HACKERS] Unfortunate choice of short switch name in pgbench

2014-02-27 Thread KONDO Mitsumasa
(2014/02/28 2:39), Tom Lane wrote: > Fabien COELHO writes: >>> Yeah, but they don't make -P take an integer argument. It's that >>> little frammish that makes this problem significant. > >> I do not see a strong case to make options with arguments case insensitive >> as a general rule. If this i

[HACKERS] What behavior is in this loop?

2014-02-27 Thread KONDO Mitsumasa
Hi, I found interesting "for" and "while" loop in WaitForWALToBecomeAvailable() in xlog.c. Can you tell me this behavior? for (;;) { ~ } while (StanbyMode) I confirmed this code is no problem in gcc compiler:) Regards, -- Mitsumasa KONDO NTT Open Source Software Center -- Sent via pgsql-hack

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2014-02-18 Thread KONDO Mitsumasa
(2014/02/17 21:44), Rajeev rastogi wrote: It got compiled successfully on Windows. Thank you for checking on Windows! It is very helpful for me. Can we allow to add three statistics? I think only adding stdev is difficult to image for user. But if there are min and max, we can image each statem

Re: [HACKERS] Exposing currentTransactionWALVolume

2014-02-17 Thread KONDO Mitsumasa
(2014/02/15 23:04), Andres Freund wrote: Hi Simon, On 2014-01-14 17:12:35 +, Simon Riggs wrote: /* - * MarkCurrentTransactionIdLoggedIfAny + * ReportTransactionInsertedWAL * - * Remember that the current xid - if it is assigned - now has been wal logged. + * Remember that the curre

Re: [HACKERS] gaussian distribution pgbench

2014-02-17 Thread KONDO Mitsumasa
(2014/02/16 7:38), Fabien COELHO wrote: I have updated the patch (v7) based on Mitsumasa latest v6: - some code simplifications & formula changes. - I've added explicit looping probability computations in comments to show the (low) looping probability of the iterative search. - I

Re: [HACKERS] gaussian distribution pgbench

2014-02-13 Thread KONDO Mitsumasa
Sorry, previos attached patch has small bug. Please use latest one. > 134 - return min + (int64) (max - min + 1) * rand; > 134 + return min + (int64)((max - min + 1) * rand); Regards, -- Mitsumasa KONDO NTT Open Source Software Center *** a/contrib/pgbench/pgbench.c --- b/contrib/pgbench/pgbench

Re: [HACKERS] gaussian distribution pgbench

2014-02-13 Thread KONDO Mitsumasa
Hi Febien, Thank you very much for your very detail and useful comments! I read your comment, I agree most of your advice:) Attached patch is fixed for your comment. That are... - Remove redundant long-option. - We can use "--gaussian=NUM -S" or "--gaussian=NUMN -N" options. - Add sentence

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2014-02-11 Thread KONDO Mitsumasa
Hi Rajeev, (2014/01/29 17:31), Rajeev rastogi wrote: No Issue, you can share me the test cases, I will take the performance report. Attached patch is supported to latest pg_stat_statements. It includes min, max, and stdev statistics. Could you run compiling test on your windows enviroments? I

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2014-01-30 Thread KONDO Mitsumasa
(2014/01/30 8:29), Tom Lane wrote: > Andrew Dunstan writes: >> I could live with just stddev. Not sure others would be so happy. > > FWIW, I'd vote for just stddev, on the basis that min/max appear to add > more to the counter update time than stddev does; you've got > this: > > + e-

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2014-01-29 Thread KONDO Mitsumasa
(2014/01/29 17:31), Rajeev rastogi wrote: On 28th January, Mitsumasa KONDO wrote: By the way, latest pg_stat_statement might affect performance in Windows system. Because it uses fflush() system call every creating new entry in pg_stat_statements, and it calls many fread() to warm file cache. It

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2014-01-29 Thread KONDO Mitsumasa
(2014/01/29 16:58), Peter Geoghegan wrote: On Tue, Jan 28, 2014 at 10:51 PM, Tom Lane wrote: KONDO Mitsumasa writes: By the way, latest pg_stat_statement might affect performance in Windows system. Because it uses fflush() system call every creating new entry in pg_stat_statements, and it

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2014-01-28 Thread KONDO Mitsumasa
(2014/01/29 15:51), Tom Lane wrote: > KONDO Mitsumasa writes: >> By the way, latest pg_stat_statement might affect performance in Windows >> system. >> Because it uses fflush() system call every creating new entry in >> pg_stat_statements, and it calls many fread()

Re: [HACKERS] drop duplicate buffers in OS

2014-01-28 Thread KONDO Mitsumasa
f->flags & BM_FADVED) && !(buf->flags & BM_JUST_DIRTIED)) (2014/01/29 8:20), Jeff Janes wrote: On Wed, Jan 15, 2014 at 10:34 AM, Robert Haas mailto:robertmh...@gmail.com>> wrote: On Wed, Jan 15, 2014 at 1:53 AM, KONDO Mitsumasa mailto:kondo.mitsum...@lab.ntt.co.jp>&

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2014-01-28 Thread KONDO Mitsumasa
(2014/01/28 15:17), Rajeev rastogi wrote: On 27th January, Mitsumasa KONDO wrote: 2014-01-26 Simon Riggs mailto:si...@2ndquadrant.com>> On 21 January 2014 19:48, Simon Riggs mailto:si...@2ndquadrant.com>> wrote: > On 21 January 2014 12:54,

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2014-01-27 Thread KONDO Mitsumasa
(2014/01/26 17:43), Mitsumasa KONDO wrote: > 2014-01-26 Simon Riggs mailto:si...@2ndquadrant.com>> > > On 21 January 2014 19:48, Simon Riggs <mailto:si...@2ndquadrant.com>> wrote: > > On 21 January 2014 12:54, KONDO Mitsumasa > <mailto:k

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2014-01-27 Thread KONDO Mitsumasa
(2014/01/23 23:18), Andrew Dunstan wrote: What is more, if the square root calculation is affecting your benchmarks, I suspect you are benchmarking the wrong thing. I run another test that has two pgbench-clients in same time, one is select-only-query and another is executing 'SELECT * pg_stat_s

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2014-01-23 Thread KONDO Mitsumasa
(2014/01/23 10:28), Peter Geoghegan wrote: On Wed, Jan 22, 2014 at 5:28 PM, KONDO Mitsumasa wrote: Oh, thanks to inform me. I think essential problem of my patch has bottle neck in sqrt() function and other division caluculation. Well, that's a pretty easy theory to test. Just stop ca

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2014-01-22 Thread KONDO Mitsumasa
(2014/01/23 12:00), Andrew Dunstan wrote: On 01/22/2014 08:28 PM, KONDO Mitsumasa wrote: (2014/01/22 22:26), Robert Haas wrote: On Wed, Jan 22, 2014 at 3:32 AM, KONDO Mitsumasa wrote: OK, Kondo, please demonstrate benchmarks that show we have <1% impact from this change. Otherwise we

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2014-01-22 Thread KONDO Mitsumasa
(2014/01/22 22:26), Robert Haas wrote: On Wed, Jan 22, 2014 at 3:32 AM, KONDO Mitsumasa wrote: OK, Kondo, please demonstrate benchmarks that show we have <1% impact from this change. Otherwise we may need a config parameter to allow the calculation. OK, testing DBT-2 now. However, er

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2014-01-22 Thread KONDO Mitsumasa
(2014/01/22 9:34), Simon Riggs wrote: AFAICS, all that has happened is that people have given their opinions and we've got almost the same identical patch, with a rush-rush comment to commit even though we've waited months. If you submit a patch, then you need to listen to feedback and be clear a

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2014-01-21 Thread KONDO Mitsumasa
Rebased patch is attached. pg_stat_statements in PG9.4dev has already changed table columns in. So I hope this patch will be committed in PG9.4dev. Regards, -- Mitsumasa KONDO NTT Open Source Software Center *** a/contrib/pg_stat_statements/pg_stat_statements--1.1--1.2.sql --- b/contrib/pg_sta

Re: [HACKERS] drop duplicate buffers in OS

2014-01-16 Thread KONDO Mitsumasa
(2014/01/16 3:34), Robert Haas wrote: On Wed, Jan 15, 2014 at 1:53 AM, KONDO Mitsumasa wrote: I create patch that can drop duplicate buffers in OS using usage_count alogorithm. I have developed this patch since last summer. This feature seems to be discussed in hot topic, so I submit it more

Re: [HACKERS] drop duplicate buffers in OS

2014-01-16 Thread KONDO Mitsumasa
(2014/01/16 21:38), Aidan Van Dyk wrote: Can we just get the backend that dirties the page to the posix_fadvice DONTNEED? No, it can remove clean page in OS file caches. Because if page is dirtied, it cause physical-disk-writing. However, it is experimental patch so it might be changed by futur

[HACKERS] drop duplicate buffers in OS

2014-01-14 Thread KONDO Mitsumasa
Hi, I create patch that can drop duplicate buffers in OS using usage_count alogorithm. I have developed this patch since last summer. This feature seems to be discussed in hot topic, so I submit it more faster than my schedule. When usage_count is high in shared_buffers, they are hard to drop fro

Re: [HACKERS] Optimize kernel readahead using buffer access strategy

2014-01-14 Thread KONDO Mitsumasa
Hi, I fix and submit this patch in CF4. In my past patch, it is significant bug which is mistaken caluculation of offset in posix_fadvise():-( However it works well without problem in pgbench. Because pgbench transactions are always random access... And I test my patch in DBT-2 benchmark. Resul

Re: [HACKERS] Optimize kernel readahead using buffer access strategy

2013-12-17 Thread KONDO Mitsumasa
(2013/12/17 21:29), Simon Riggs wrote: These are interesting results. Good research. Thanks! They also show that the benefit of this is very specific to the exact task being performed. I can't see any future for a setting that applies to everything or nothing. We must be more selective. This

Re: [HACKERS] pg_rewarm status

2013-12-17 Thread KONDO Mitsumasa
(2013/12/18 5:33), Robert Haas wrote: Sounds like it might be worth dusting the patch off again... I'd like to request you to add all_index option and usage_count option. When all_index option is selected, all index become rewarm nevertheless user doesn't input relation name. And usage_count op

Re: [HACKERS] Optimize kernel readahead using buffer access strategy

2013-12-17 Thread KONDO Mitsumasa
Hi, I fixed the patch to improve followings. - Can compile in MacOS. - Change GUC name enable_kernel_readahead to readahead_strategy. - Change POSIX_FADV_SEQUNENTIAL to POISX_FADV_NORMAL when we select sequential access strategy, this reason is later... I tested simple two access pate

Re: [HACKERS] Optimize kernel readahead using buffer access strategy

2013-12-12 Thread KONDO Mitsumasa
(2013/12/12 9:30), Claudio Freire wrote: On Wed, Dec 11, 2013 at 3:14 AM, KONDO Mitsumasa wrote: enable_readahead=os|fadvise with os = on, fadvise = off Hmm. fadvise is method and is not a purpose. So I consider another idea of this GUC. Yeah, I was thinking of opening the door for

Re: [HACKERS] Time-Delayed Standbys

2013-12-12 Thread KONDO Mitsumasa
(2013/12/12 18:09), Simon Riggs wrote: On 9 December 2013 10:54, KONDO Mitsumasa wrote: (2013/12/09 19:35), Pavel Stehule wrote: 2013/12/9 KONDO Mitsumasa mailto:kondo.mitsum...@lab.ntt.co.jp>> Hi Fabrízio, I test your v4 patch, and send your review comments. * Fi

Re: [HACKERS] Time-Delayed Standbys

2013-12-12 Thread KONDO Mitsumasa
(2013/12/12 7:23), Fabrízio de Royes Mello wrote: On Wed, Dec 11, 2013 at 7:47 PM, Andres Freund * hot_standby=off: Makes delay useable with wal_level=archive (and thus > a lower WAL volume) > * standby_mode=off: Configurations that use tools like pg_standby and > similar simply don't nee

Re: [HACKERS] Time-Delayed Standbys

2013-12-10 Thread KONDO Mitsumasa
(2013/12/10 18:38), Andres Freund wrote: "master PITR"? What's that? All PITR is based on recovery.conf and thus not really a "master"? "master PITR" is PITR with "standby_mode = off". It's just recovery from basebackup. They have difference between "master PITR" and "standby" that the former w

Re: [HACKERS] Optimize kernel readahead using buffer access strategy

2013-12-10 Thread KONDO Mitsumasa
(2013/12/10 22:55), Claudio Freire wrote: On Tue, Dec 10, 2013 at 5:03 AM, KONDO Mitsumasa wrote: I revise this patch and re-run performance test, it can work collectry in Linux and no complile wanings. I add GUC about enable_kernel_readahead option in new version. When this GUC is on(default

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-10 Thread KONDO Mitsumasa
(2013/12/11 10:25), Tom Lane wrote: > Jeff Janes writes: >> On Tue, Dec 3, 2013 at 11:39 PM, Claudio Freire >> wrote: >>> Problem is, Postgres relies on a working kernel cache for checkpoints. >>> Checkpoint logic would have to be heavily reworked to account for an >>> impaired kernel cache. > >

Re: [HACKERS] Optimize kernel readahead using buffer access strategy

2013-12-09 Thread KONDO Mitsumasa
Hi, I revise this patch and re-run performance test, it can work collectry in Linux and no complile wanings. I add GUC about enable_kernel_readahead option in new version. When this GUC is on(default), it works in POSIX_FADV_NORMAL which is general readahead in OS. And when it is off, it works

Re: [HACKERS] Time-Delayed Standbys

2013-12-09 Thread KONDO Mitsumasa
(2013/12/09 20:29), Andres Freund wrote: On 2013-12-09 19:51:01 +0900, KONDO Mitsumasa wrote: Add my comment. We have to consider three situations. 1. PITR 2. replication standby 3. replication standby with restore_command I think this patch cannot delay in 1 situation. Why? I have three

Re: [HACKERS] Time-Delayed Standbys

2013-12-09 Thread KONDO Mitsumasa
(2013/12/09 19:35), Pavel Stehule wrote: 2013/12/9 KONDO Mitsumasa mailto:kondo.mitsum...@lab.ntt.co.jp>> Hi Fabrízio, I test your v4 patch, and send your review comments. * Fix typo > 49 -# commited transactions from the master, specify a recovery time delay.

Re: [HACKERS] Time-Delayed Standbys

2013-12-09 Thread KONDO Mitsumasa
(2013/12/09 19:36), KONDO Mitsumasa wrote: * Problem 1 I read your wittened document. There is "PITR has not affected". However, when I run PITR with min_standby_apply_delay=300, it cannot start server. The log is under following. [mitsu-ko@localhost postgresql]$ bin/pg_ctl -D d

Re: [HACKERS] Time-Delayed Standbys

2013-12-09 Thread KONDO Mitsumasa
Hi Fabrízio, I test your v4 patch, and send your review comments. * Fix typo > 49 -# commited transactions from the master, specify a recovery time delay. > 49 +# committed transactions from the master, specify a recovery time delay. * Fix white space > 177 - if (secs <= 0 && micr

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-08 Thread KONDO Mitsumasa
(2013/12/05 23:42), Greg Stark wrote: On Thu, Dec 5, 2013 at 8:35 AM, KONDO Mitsumasa wrote: Yes. And using something efficiently DirectIO is more difficult than BufferedIO. If we change write() flag with direct IO in PostgreSQL, it will execute hardest ugly randomIO. Using DirectIO presumes

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-05 Thread KONDO Mitsumasa
(2013/12/04 16:39), Claudio Freire wrote: On Wed, Dec 4, 2013 at 4:28 AM, Tatsuo Ishii wrote: Can we avoid the Linux kernel problem by simply increasing our shared buffer size, say up to 80% of memory? It will be swap more easier. Is that the case? If the system has not enough memory, the ke

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-03 Thread KONDO Mitsumasa
(2013/12/04 11:28), Tatsuo Ishii wrote: Magnus Hagander writes: On Tue, Dec 3, 2013 at 11:44 PM, Josh Berkus wrote: Would certainly be nice. Realistically, getting good automated performace tests will require paying someone like Greg S., Mark or me for 6 solid months to develop them, since w

Re: [HACKERS] Time-Delayed Standbys

2013-12-03 Thread KONDO Mitsumasa
(2013/12/04 4:00), Andres Freund wrote: On 2013-12-03 13:46:28 -0500, Robert Haas wrote: On Tue, Dec 3, 2013 at 12:36 PM, Fabrízio de Royes Mello wrote: On Tue, Dec 3, 2013 at 2:33 PM, Christian Kruse wrote: Hi Fabrizio, looks good to me. I did some testing on 9.2.4, 9.2.5 and HEAD. It app

Re: [HACKERS] Time-Delayed Standbys

2013-12-03 Thread KONDO Mitsumasa
(2013/11/30 5:34), Fabrízio de Royes Mello wrote: On Fri, Nov 29, 2013 at 5:49 AM, KONDO Mitsumasa mailto:kondo.mitsum...@lab.ntt.co.jp>> wrote: > * Problem1 > Your patch does not code recovery.conf.sample about recovery_time_delay. > Please add it. Fixed. OK. It s

Re: [HACKERS] Time-Delayed Standbys

2013-11-28 Thread KONDO Mitsumasa
Hi Royes, I'm sorry for my late review... I feel potential of your patch in PG replication function, and it might be something useful for all people. I check your patch and have some comment for improvement. I haven't executed detail of unexpected sutuation yet. But I think that under followi

Re: [HACKERS] Logging WAL when updating hintbit

2013-11-18 Thread KONDO Mitsumasa
(2013/11/15 19:27), Sawada Masahiko wrote: On Thu, Nov 14, 2013 at 7:51 PM, Florian Weimer wrote: On 11/14/2013 07:02 AM, Sawada Masahiko wrote: I attached patch adds new wal_level 'all'. Shouldn't this be a separate setting? It's useful for storage which requires rewriting a partially wr

Re: [HACKERS] Improvement of pg_stat_statement usage about buffer hit ratio

2013-11-18 Thread KONDO Mitsumasa
(2013/11/19 12:03), Peter Geoghegan wrote: On Mon, Nov 18, 2013 at 6:12 PM, KONDO Mitsumasa wrote: I confirmed that Itagaki-san and Mr Cerdic disscution. He said that raw values be just simple. However, were his changes just simple? I cannot understand his aesthetics sense and also you, too

Re: [HACKERS] Improvement of pg_stat_statement usage about buffer hit ratio

2013-11-18 Thread KONDO Mitsumasa
(2013/11/19 11:12), KONDO Mitsumasa wrote: (2013/11/19 3:56), Peter Geoghegan wrote: On Mon, Nov 18, 2013 at 10:49 AM, Fujii Masao wrote: The same idea was proposed before but not committed because Itagaki thought that pg_stat_statements view should report only raw values. Please read the

Re: [HACKERS] Improvement of pg_stat_statement usage about buffer hit ratio

2013-11-18 Thread KONDO Mitsumasa
(2013/11/19 3:56), Peter Geoghegan wrote: On Mon, Nov 18, 2013 at 10:49 AM, Fujii Masao wrote: The same idea was proposed before but not committed because Itagaki thought that pg_stat_statements view should report only raw values. Please read the following thread. I have the same feeling with h

Re: [HACKERS] Improvement of pg_stat_statement usage about buffer hit ratio

2013-11-18 Thread KONDO Mitsumasa
(2013/11/18 20:16), Haribabu kommi wrote: On 18 October 2013 13:35 KONDO Mitsumasa wrote: This patch conflicts pg_stat_statement_min_max_exectime patch which I submitted, and pg_stat_statement_min_max_exectime patch also adds new columns which are min_time and max_time. So I'd like to chan

Re: [HACKERS] Optimize kernel readahead using buffer access strategy

2013-11-17 Thread KONDO Mitsumasa
(2013/11/18 11:25), Claudio Freire wrote: On Sun, Nov 17, 2013 at 11:02 PM, KONDO Mitsumasa wrote: However, my patch is on the way and needed to more improvement. I am going to add method of controlling readahead by GUC, for user can freely select readahed parameter in their transactions

Re: [HACKERS] Optimize kernel readahead using buffer access strategy

2013-11-17 Thread KONDO Mitsumasa
(2013/11/15 13:48), Claudio Freire wrote: On Thu, Nov 14, 2013 at 11:13 PM, KONDO Mitsumasa I use CentOS 6.4 which kernel version is 2.6.32-358.23.2.el6.x86_64 in this test. That's close to the kernel version I was using, so you should see the same effect. OK. You proposed readahead ma

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2013-11-14 Thread KONDO Mitsumasa
(2013/11/15 2:09), Fujii Masao wrote: Agreed. Could you tell me your agreed reason? I am sorry that I suspect you doesn't understand this disccusion enough:-( Regards, -- Mitsumasa KONDO NTT Open Source Software Ceter -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To m

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2013-11-14 Thread KONDO Mitsumasa
(2013/11/15 11:31), Peter Geoghegan wrote: On Thu, Nov 14, 2013 at 6:28 PM, KONDO Mitsumasa wrote: It is confirmation just to make sure, does "this patch" mean my patch? I agree with you about not adding another lock implementation. It will becomes overhead. Yes, I referred to you

Re: [HACKERS] Optimize kernel readahead using buffer access strategy

2013-11-14 Thread KONDO Mitsumasa
(2013/11/15 11:17), Peter Geoghegan wrote: On Thu, Nov 14, 2013 at 6:18 PM, KONDO Mitsumasa wrote: I will fix it. Could you tell me your Mac OS version and gcc version? I have only mac book air with Maverick OS(10.9). I have an idea that Mac OSX doesn't have posix_fadvise at all. Didn&

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2013-11-14 Thread KONDO Mitsumasa
(2013/11/14 7:11), Peter Geoghegan wrote: On Wed, Oct 23, 2013 at 8:52 PM, Alvaro Herrera wrote: Hmm, now if we had portable atomic addition, so that we could spare the spinlock ... And adding a histogram or min/max for something like execution time isn't an approach that can be made to work f

Re: [HACKERS] Optimize kernel readahead using buffer access strategy

2013-11-14 Thread KONDO Mitsumasa
(2013/11/15 2:03), Fujii Masao wrote: On Thu, Nov 14, 2013 at 9:09 PM, KONDO Mitsumasa wrote: Hi, I create a patch that is improvement of disk-read and OS file caches. It can optimize kernel readahead parameter using buffer access strategy and posix_fadvice() in various disk-read situations

Re: [HACKERS] Optimize kernel readahead using buffer access strategy

2013-11-14 Thread KONDO Mitsumasa
Hi Claudio, (2013/11/14 22:53), Claudio Freire wrote: On Thu, Nov 14, 2013 at 9:09 AM, KONDO Mitsumasa wrote: I create a patch that is improvement of disk-read and OS file caches. It can optimize kernel readahead parameter using buffer access strategy and posix_fadvice() in various disk-read

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2013-11-14 Thread KONDO Mitsumasa
Oh! Sorry... I forgot to attach my latest patch. Regards, -- Mitsumasa KONDO NTT Open Source Software Center diff --git a/contrib/pg_stat_statements/pg_stat_statements--1.1--1.2.sql b/contrib/pg_stat_statements/pg_stat_statements--1.1--1.2.sql new file mode 100644 index 000..929d623 --- /dev/n

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2013-11-14 Thread KONDO Mitsumasa
(2013/10/21 20:17), KONDO Mitsumasa wrote: > (2013/10/18 22:21), Andrew Dunstan wrote: >> If we're going to extend pg_stat_statements, even more than min and max >> I'd like to see the standard deviation in execution time. > OK. I do! I am making some other patc

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2013-10-22 Thread KONDO Mitsumasa
Hi All, (2013/10/22 22:26), Stephen Frost wrote: * Dimitri Fontaine (dimi...@2ndquadrant.fr) wrote: In our case, what I keep experiencing with tuning queries is that we have like 99% of them running under acceptable threshold and 1% of them taking more and more time. This is usually described

Re: [HACKERS] Compression of full-page-writes

2013-10-22 Thread KONDO Mitsumasa
(2013/10/22 12:52), Fujii Masao wrote: On Tue, Oct 22, 2013 at 12:47 PM, Amit Kapila wrote: On Mon, Oct 21, 2013 at 4:40 PM, KONDO Mitsumasa wrote: (2013/10/19 14:58), Amit Kapila wrote: On Tue, Oct 15, 2013 at 11:41 AM, KONDO Mitsumasa wrote: In general, my thinking is that we should

Re: [HACKERS] Add min and max execute statement time in pg_stat_statement

2013-10-21 Thread KONDO Mitsumasa
(2013/10/18 22:21), Andrew Dunstan wrote: > If we're going to extend pg_stat_statements, even more than min and max > I'd like to see the standard deviation in execution time. OK. I do! I am making some other patches, please wait more! Regards, -- Mitsumasa KONDO NTT Open Source Software Center.;

Re: [HACKERS] Compression of full-page-writes

2013-10-21 Thread KONDO Mitsumasa
(2013/10/19 14:58), Amit Kapila wrote: > On Tue, Oct 15, 2013 at 11:41 AM, KONDO Mitsumasa > wrote: > I think in general also snappy is mostly preferred for it's low CPU > usage not for compression, but overall my vote is also for snappy. I think low CPU usage is the best import

Re: [HACKERS] Who is pgFoundery administrator?

2013-10-21 Thread KONDO Mitsumasa
(2013/10/02 18:57), Michael Paquier wrote: wrote: Who is pgFoundery administrator or board member now? I would like to send e-mail them. At least, it does not have information and support page in pgFoundery homepage. Why don't you consider github as a potential solution? It is because github

[HACKERS] Improvement of pg_stat_statement usage about buffer hit ratio

2013-10-18 Thread KONDO Mitsumasa
Hi, I submit improvement of pg_stat_statement usage patch in CF3. In pg_stat_statement, I think buffer hit ratio is very important value. However, it is difficult to calculate it, and it need complicated SQL. This patch makes it more simple usage and documentation. > -bench=# SELECT query, call

[HACKERS] Add min and max execute statement time in pg_stat_statement

2013-10-18 Thread KONDO Mitsumasa
I submit patch adding min and max execute statement time in pg_stat_statement in next CF. pg_stat_statement have execution time, but it is average execution time and does not provide detail information very much. So I add min and max execute statement time in pg_stat_statement columns. Usage is al

Re: [HACKERS] Compression of full-page-writes

2013-10-15 Thread KONDO Mitsumasa
(2013/10/15 22:01), k...@rice.edu wrote: Google's lz4 is also a very nice algorithm with 33% better compression performance than snappy and 2X the decompression performance in some benchmarks also with a bsd license: https://code.google.com/p/lz4/ If we judge only performance, we will select lz

Re: [HACKERS] Compression of full-page-writes

2013-10-14 Thread KONDO Mitsumasa
(2013/10/15 13:33), Amit Kapila wrote: Snappy is good mainly for un-compressible data, see the link below: http://www.postgresql.org/message-id/CAAZKuFZCOCHsswQM60ioDO_hk12tA7OG3YcJA8v=4yebmoa...@mail.gmail.com This result was gotten in ARM architecture, it is not general CPU. Please see detail

Re: [HACKERS] Release note fix for timeline item

2013-10-14 Thread KONDO Mitsumasa
Sorry for my reply late... (2013/10/08 23:26), Bruce Momjian wrote: > First, I want to apologize for not completing the release notes earlier > so that others could review them. I started working on the release > notes on Friday, but my unfamiliarity with the process and fear of > making a mista

Re: [HACKERS] Compression of full-page-writes

2013-10-14 Thread KONDO Mitsumasa
(2013/10/13 0:14), Amit Kapila wrote: On Fri, Oct 11, 2013 at 10:36 PM, Andres Freund wrote: But maybe pglz is just not a good fit for this, it really isn't a very good algorithm in this day and aage. +1. This compression algorithm is needed more faster than pglz which is like general compress

Re: [HACKERS] Compression of full-page-writes

2013-10-08 Thread KONDO Mitsumasa
(2013/10/08 20:13), Haribabu kommi wrote: I chosen the sync_commit=off mode because it generates more tps, thus it increases the volume of WAL. I did not think to there. Sorry... I will test with sync_commit=on mode and provide the test results. OK. Thanks! -- Mitsumasa KONDO NTT Open Sourc

Re: [HACKERS] Compression of full-page-writes

2013-10-08 Thread KONDO Mitsumasa
Hi, I tested dbt-2 benchmark in single instance and synchronous replication. Unfortunately, my benchmark results were not seen many differences... * Test server Server: HP Proliant DL360 G7 CPU:Xeon E5640 2.66GHz (1P/4C) Memory: 18GB(PC3-10600R-9) Disk: 146GB(15k)*4 RAID1+0

Re: [HACKERS] Compression of full-page-writes

2013-10-08 Thread KONDO Mitsumasa
(2013/10/08 17:33), Haribabu kommi wrote: The checkpoint_timeout and checkpoint_segments are increased to make sure no checkpoint happens during the test run. Your setting is easy occurred checkpoint in checkpoint_segments = 256. I don't know number of disks in your test server, in my test serv

Re: [HACKERS] Who is pgFoundery administrator?

2013-10-02 Thread KONDO Mitsumasa
(2013/10/02 17:37), KONDO Mitsumasa wrote: > I want to submit new project in pgFoundery project. Our new project was approved yesterday! Thanks very much for pgFoundery crew. Regards, -- Mitsumasa KONDO NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hack

[HACKERS] Who is pgFoundery administrator?

2013-10-02 Thread KONDO Mitsumasa
Hi, I want to submit new project in pgFoundery project. I submitted new project which is WAL archive copy tool with directIO method in pgFoundery homepage 2 weeks ago, but it does not have approved and responded at all:-( Who is pgFoundery administrator or board member now? I would like to send

Re: [HACKERS] Compression of full-page-writes

2013-09-30 Thread KONDO Mitsumasa
(2013/09/30 13:55), Amit Kapila wrote: On Mon, Sep 30, 2013 at 10:04 AM, Fujii Masao wrote: Yep, please! It's really helpful! OK! I test with single instance and synchronous replication constitution. By the way, you posted patch which is sync_file_range() WAL writing method in 3 years ago. I

Re: [HACKERS] Compression of full-page-writes

2013-09-29 Thread KONDO Mitsumasa
Hi Fujii-san, (2013/09/30 12:49), Fujii Masao wrote: > On second thought, the patch could compress WAL very much because I used pgbench. I will do the same measurement by using another benchmark. If you hope, I can test this patch in DBT-2 benchmark in end of this week. I will use under follow

Re: [HACKERS] gaussian distribution pgbench

2013-09-29 Thread KONDO Mitsumasa
(2013/09/27 5:29), Peter Eisentraut wrote: > This patch no longer applies. I will try to create this patch in next commit fest. If you have nice idea, please send me! Regards, -- Mitsumasa KONDO NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.or

Re: [HACKERS] gaussian distribution pgbench

2013-09-29 Thread KONDO Mitsumasa
Sorry for my delay reply. Since I have had vacation last week, I replied from gmail. However, it was stalled post to pgsql-hackers:-( (2013/09/21 7:54), Fabien COELHO wrote: However this pattern induces stronger cache effects which are maybe not too realistic, because neighboring keys in the mi

Re: [HACKERS] gaussian distribution pgbench

2013-09-29 Thread KONDO Mitsumasa
Sorry for my delay reply. Since I have had vacation last week, I replyed from gmail. However, it was stalled post to pgsql-hackers:-( (2013/09/21 6:05), Kevin Grittner wrote: > You had accidentally added to the CF In Progress. Oh, I had completely mistook this CF schedule :-) Maybe, Horiguchi-san

Re: [HACKERS] 9.4 regression

2013-09-05 Thread KONDO Mitsumasa
(2013/09/05 0:04), Andres Freund wrote: > I'd vote for adding zeroing *after* the fallocate() first. +1, with FALLOC_FL_KEEP_SIZE flag. At least, fallocate with FALLOC_FL_KEEP_SIZE flag is faster than nothing in my developing sorted checkpoint. I adopted it to relation file, so I don't know abo

[HACKERS] Add pgbench option: CHECKPOINT before starting benchmark

2013-08-30 Thread KONDO Mitsumasa
Hi, I add checkpoint option to pgbench. pgbench is simple and useful benchmark for every user. However, result of benchmark greatly changes by some situations which are in executing checkpoint, number of dirty buffers in share_buffers, and so on. For such a problem, it is custom to carry out a ch

  1   2   >