Re: [HACKERS] Performance degradation in commit ac1d794

2016-03-20 Thread Andres Freund
On 2016-03-17 09:01:36 -0400, Robert Haas wrote: > > * Right now the caller has to allocate the WaitEvents he's waiting for > > locally (likely on the stack), but we also could allocate them as part > > of the WaitEventSet. Not sure if that'd be a benefit. > > I'm not seeing this. What do you

Re: [HACKERS] [COMMITTERS] pgsql: Improve memory management for external sorts.

2016-03-20 Thread Andres Freund
On 2016-03-18 14:40:33 -0400, Robert Haas wrote: > > Oops. Thanks for the report. Does this fix it? > > Trying again to attach the patch. Yes, that removes the warning, and looks correct. Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To mak

Re: [HACKERS] Performance degradation in commit ac1d794

2016-03-20 Thread Andres Freund
On 2016-03-21 01:31:30 +1300, Thomas Munro wrote: > I couldn't get the second patch to apply for some reason, Weird? Even efter appying the first one first? > but I have been trying out your "latch" branch on some different OSs > and porting some code that does a bunch of waiting on many sockets

Re: [HACKERS] Performance degradation in commit ac1d794

2016-03-20 Thread Andres Freund
On 2016-03-18 05:56:41 -0400, Robert Haas wrote: > >> 0 at the top of the loop and skip it forthwith if so. > > > > You mean in WaitEventSetWait()? There's > > else if (rc == 0) > > { > > break; > > } > > which is the timeout c

Re: [HACKERS] Performance degradation in commit ac1d794

2016-03-20 Thread Andres Freund
On March 21, 2016 5:12:38 AM GMT+01:00, Amit Kapila wrote: >The article pointed by you justifies that the way ResetEvent is done by >patch is correct. I am not sure, but you can weigh, if there is a need >of >comment so that if we want enhance this part of code (or want to write >something si

Re: [HACKERS] Performance degradation in commit ac1d794

2016-03-20 Thread Andres Freund
Hi, On 2016-03-21 11:52:43 +1300, Thomas Munro wrote: > The contract that I invented here is that an async-aware parent node > can ask any child node "are you ready?" and get back various answers > including an fd which means please don't call ExecProcNode until this > fd is ready to read. But a

[HACKERS] latch usage and postmaster death

2016-03-21 Thread Andres Freund
es between the two will possibly be ignored. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Performance degradation in commit ac1d794

2016-03-21 Thread Andres Freund
ese after some msvc animals turned green. We might want to iterate on the API in some parts, but this will fix the regression, and imo looks good. Regards, Andres >From 72e2d21c1249b674496f97cd6009c0bda62f6b4d Mon Sep 17 00:00:00 2001 From: Andres Freund Date: Mon, 21 Mar 2016 09:56:

Re: [HACKERS] Performance degradation in commit ac1d794

2016-03-21 Thread Andres Freund
Hi, I've pushed the new API. We might want to use it in more places... On 2015-12-25 20:08:15 +0300, Васильев Дмитрий wrote: > I suddenly found commit ac1d794 gives up to 3 times performance degradation. > > I tried to run pgbench -s 1000 -j 48 -c 48 -S -M prepared on 70 CPU-core > machine: > co

Re: [HACKERS] fd.c: flush data problems on osx

2016-03-21 Thread Andres Freund
only OSX, but I didn’t tried). > So i’ve skipped mmap for dirs and now restore happens without > warnings. Also I’ve fixed wrong error check that was in previous > version of patch. Hm. I think we should rather just skip calling pg_flush_data in the directory case, that very likely isn&#

Re: [HACKERS] Applying logical replication changes by more than one process

2016-03-21 Thread Andres Freund
On March 21, 2016 2:08:54 PM GMT+01:00, Petr Jelinek wrote: >On 21/03/16 13:44, Konstantin Knizhnik wrote: >> >> >> On 21.03.2016 15:10, Petr Jelinek wrote: >>> Hi, >>> >>> On 19/03/16 11:46, Konstantin Knizhnik wrote: Hi, I am trying to use logical replication mechanism in imple

Re: [HACKERS] Applying logical replication changes by more than one process

2016-03-21 Thread Andres Freund
On 2016-03-21 14:18:27 +0100, Petr Jelinek wrote: > On 21/03/16 14:15, Andres Freund wrote: > >>Only when the origin is actually setup for the current session. You > >>need > >>to call the replorigin_advance yourself from your apply code. > > > &g

Re: [HACKERS] checkpointer continuous flushing

2016-03-22 Thread Andres Freund
nsactions were scheduled. Again, the new code is winning. Thanks for running these tests! I think this shows that we're in a good shape, and that the commits succeeded in what they were attempting. Very glad to hear that. WRT tablespaces: What I'm planning to do, unless somebody has

Re: [HACKERS] checkpointer continuous flushing

2016-03-22 Thread Andres Freund
On 2016-03-22 10:48:20 +0100, Tomas Vondra wrote: > Hi, > > On 03/22/2016 10:44 AM, Fabien COELHO wrote: > > > > > 1) regular-latency.png > >>> > >>>I'm wondering whether it would be clearer if the percentiles > >>>where relative to the largest sample, not to itself, so that the > >>>figures f

Re: [HACKERS] checkpointer continuous flushing

2016-03-22 Thread Andres Freund
On 2016-03-22 10:52:55 +0100, Fabien COELHO wrote: > To emphasize potential bad effects without having to build too large a host > and involve too many table spaces, I would suggest to reduce significantly > the "checkpoint_flush_after" setting while running these tests. Meh, that completely disto

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

2016-03-22 Thread Andres Freund
> * clog group member */ > + Man, we're surely bloating PGPROC at a prodigious rate. That's my first pass over the code itself. Hm. Details aside, what concerns me most is that the whole group mechanism, as implemented, only works als long as

Re: [HACKERS] Proposal: Generic WAL logical messages

2016-03-22 Thread Andres Freund
_ID, "LogicalMessage", logicalmsg_redo, > logicalmsg_desc, logicalmsg_identify, NULL, NULL) Did you consider doing this via the standby rmgr instead? > +typedef struct xl_logical_message > +{ > + booltransactional; /* is > message transactional? */ > + size_t prefix_size;/* > length of prefix */ > + size_t message_size; /* size > of the message */ > + charmessage[FLEXIBLE_ARRAY_MEMBER]; /* message including > the null > + > * terminated prefx of length > + > * prefix_size */ > +} xl_logical_message; > "prefx". Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Timeline following for logical slots

2016-03-22 Thread Andres Freund
On 2016-03-15 21:04:12 +0800, Craig Ringer wrote: > Thanks very much for the review. Are you planning to update the patch? - Andres -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

2016-03-22 Thread Andres Freund
On 2016-03-22 18:19:48 +0530, Amit Kapila wrote: > > I'm actually rather unconvinced that it's all that common that all > > subtransactions are on one page. If you have concurrency - otherwise > > there'd be not much point in this patch - they'll usually be heavily > > interleaved, no? You can arg

Re: [HACKERS] NOT EXIST for PREPARE

2016-03-22 Thread Andres Freund
Hi, On 2016-03-22 12:41:43 +0300, Yury Zhuravlev wrote: > Do I understand correctly the only way know availability PREPARE it will > appeal to pg_prepared_statements? > I think this is not a good practice. In some cases, we may not be aware of > the PREPARE made (pgpool). Moreover, it seems popul

Re: [HACKERS] Proposal: Generic WAL logical messages

2016-03-22 Thread Andres Freund
On 2016-03-22 14:03:06 +0100, Petr Jelinek wrote: > On 22/03/16 12:47, Andres Freund wrote: > >On 2016-03-21 18:10:55 +0100, Petr Jelinek wrote: > > > >>+ > >>+ > >>+ Generic Message Callback > >>+ > >>+ > >>+

Re: [HACKERS] NOT EXIST for PREPARE

2016-03-22 Thread Andres Freund
On 2016-03-22 16:51:26 +0300, Yury Zhuravlev wrote: > Andres Freund wrote: > >you'd still need to compare the > >statement somehow > > You right, I think about that as syntax sugar. Maybe with some performance > increase but hardly. We can save on a round trip. If an

Re: [HACKERS] NOT EXIST for PREPARE

2016-03-22 Thread Andres Freund
On 2016-03-22 09:37:15 -0500, Merlin Moncure wrote: > On Tue, Mar 22, 2016 at 8:01 AM, Andres Freund wrote: > > Hi, > > On 2016-03-22 12:41:43 +0300, Yury Zhuravlev wrote: > >> Do I understand correctly the only way know availability PREPARE it will > >> appeal

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

2016-03-22 Thread Andres Freund
On 2016-03-22 10:40:28 -0400, Robert Haas wrote: > On Tue, Mar 22, 2016 at 6:52 AM, Andres Freund wrote: > > I'm actually rather unconvinced that it's all that common that all > > subtransactions are on one page. If you have concurrency - otherwise > > there&#x

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

2016-03-23 Thread Andres Freund
On 2016-03-23 12:33:22 +0530, Amit Kapila wrote: > On Wed, Mar 23, 2016 at 12:26 PM, Amit Kapila > wrote: > > > > On Tue, Mar 22, 2016 at 4:22 PM, Andres Freund wrote: > > > > > > > > > I think it's worthwhile to create a benchmark that doe

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

2016-03-23 Thread Andres Freund
On 2016-03-23 21:43:41 +0100, Andres Freund wrote: > I'm playing around with SELECT txid_current(); right now - that should > be about the most specific load for setting clog bits. Or so I thought. In my testing that showed just about zero performance difference between the patch and

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

2016-03-23 Thread Andres Freund
Hi, On 2016-03-24 01:10:55 +0100, Andres Freund wrote: > I'm afraid that this patch might be putting bandaid on some of the > absolutely worst cases, without actually addressing the core > problem. Simon's patch in [1] seems to come closer addressing that > (which I d

[HACKERS] Re: [COMMITTERS] pgsql: Move each SLRU's lwlocks to a separate tranche.

2016-03-25 Thread Andres Freund
that should be fixed? I guess it'd be good to avoid duplicating the code for aligning, so maybe we ought to add a ShmemAllocAligned or something? Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Performance degradation in commit 6150a1b0

2016-03-25 Thread Andres Freund
On 2016-03-25 09:29:34 +0530, Amit Kapila wrote: > > 2. Secondly, i can see that the BufferDesc structure padding is 64 bytes > however the PG CACHE LINE ALIGNMENT is 128 bytes. Also, after changing the > BufferDesc structure padding size to 128 bytes along with the changes > mentioned in above poi

Re: [HACKERS] [COMMITTERS] pgsql: Move each SLRU's lwlocks to a separate tranche.

2016-03-25 Thread Andres Freund
On March 25, 2016 1:04:13 PM GMT+01:00, Robert Haas wrote: >On Fri, Mar 25, 2016 at 3:05 AM, Andres Freund >wrote: >> On 2015-11-12 19:59:54 +, Robert Haas wrote: >>> Move each SLRU's lwlocks to a separate tranche. >>> >>> This makes it signi

Re: [HACKERS] [COMMITTERS] pgsql: Move each SLRU's lwlocks to a separate tranche.

2016-03-25 Thread Andres Freund
On March 25, 2016 2:48:00 PM GMT+01:00, Robert Haas wrote: >On Fri, Mar 25, 2016 at 9:11 AM, Andres Freund >wrote: >> On March 25, 2016 1:04:13 PM GMT+01:00, Robert Haas > wrote: >>>On Fri, Mar 25, 2016 at 3:05 AM, Andres Freund >>>wrote: >>>> On

Re: [HACKERS] Move PinBuffer and UnpinBuffer to atomics

2016-03-27 Thread Andres Freund
On 2016-03-25 23:02:11 +0530, Dilip Kumar wrote: > On Fri, Mar 25, 2016 at 8:09 PM, Alexander Korotkov < > a.korot...@postgrespro.ru> wrote: > > > Could anybody run benchmarks? Feature freeze is soon, but it would be > > *very nice* to fit it into 9.6 release cycle, because it greatly improves >

Re: [HACKERS] Move PinBuffer and UnpinBuffer to atomics

2016-03-27 Thread Andres Freund
st of cases slightly outperforms v3. What commit did you base these tests on? I guess something recent, after 98a64d0bd? > I'm going to do some code cleanup of v5 in Monday Ok, I'll try to do a review and possibly commit after that. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Performance degradation in commit 6150a1b0

2016-03-27 Thread Andres Freund
Hi, On 2016-03-27 02:34:32 +0530, Ashutosh Sharma wrote: > As mentioned in my earlier mail i was not able to apply > *pinunpin-cas-5.patch* on commit *6150a1b0, That's not surprising; that's pretty old. > *therefore i thought of applying it on the latest commit and i was > able to do it successf

Re: [HACKERS] Move PinBuffer and UnpinBuffer to atomics

2016-03-27 Thread Andres Freund
On 2016-03-27 17:45:52 +0530, Dilip Kumar wrote: > On Sun, Mar 27, 2016 at 5:37 PM, Andres Freund wrote: > > > On what hardware did you run these tests? > > > IBM POWER 8 MACHINE. > > Architecture: ppc64le > Byte Order:Little Endian &g

Re: [HACKERS] GinPageIs* don't actually return a boolean

2016-03-27 Thread Andres Freund
fixed the macros instead of the callsites) to all branches. I plan to commit the larger patch (which has grown since last posting it) after the minor releases; it's somewhat large and has a lot of conflicts. This way at least the confirmed issue is fixed in the next set of minor releases. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] Automatically add -Wold-style-definition?

2016-03-27 Thread Andres Freund
ke; and it doesn't have false positives. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Automatically add -Wold-style-definition?

2016-03-27 Thread Andres Freund
On 2016-03-27 17:16:11 -0400, Tom Lane wrote: > Andres Freund writes: > > Does somebody see a reason not to automatically detect and use > > -Wold-style-definition? > > +1 ... unless we have some that are that way intentionally, which > I kinda doubt, but you could soon f

Re: [HACKERS] pg_rewind just doesn't fsync *anything*?

2016-03-27 Thread Andres Freund
On 2016-03-09 19:43:52 -0800, Andres Freund wrote: > Hi, > > how come that the only comment in pg_rewind about fsyncing is ' > void > close_target_file(void) > { > ... > /* fsync? */ > } > > Isn't that a bit, uh, minimal for a utility that&

Re: [HACKERS] silent data loss with ext4 / all current versions

2016-03-27 Thread Andres Freund
that's actually not unlikely after a crash. Even with an fsync that's not guaranteed not to happen, but the chance of it is much lower. I'm too tired to push this at the eleventh hour. Besides a heavily revised patch, backpatching will likely include a number of conflicts. If someb

[HACKERS] backup tools ought to ensure created backups are durable

2016-03-27 Thread Andres Freund
7;s pretty much unaceptable. There's cases where we can't ensure durability (i.e. pg_dump | gzip > file), but it's out of our hands in that case. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscriptio

Re: [HACKERS] Move PinBuffer and UnpinBuffer to atomics

2016-03-28 Thread Andres Freund
On 2016-03-28 11:48:46 +0530, Dilip Kumar wrote: > On Sun, Mar 27, 2016 at 5:48 PM, Andres Freund wrote: > > > > > What's sizeof(BufferDesc) after applying these patches? It should better > > be <= 64... > > > > It is 72. Ah yes, miscalculated the re

Re: [HACKERS] backup tools ought to ensure created backups are durable

2016-03-28 Thread Andres Freund
On 2016-03-28 11:35:57 +0200, Magnus Hagander wrote: > On Mon, Mar 28, 2016 at 3:11 AM, Michael Paquier > wrote: > > > On Mon, Mar 28, 2016 at 8:30 AM, Andres Freund wrote: > > > As pointed out in > > > > > http://www.postgresql.org/message-id/20160327232

Re: [HACKERS] pg_xlogdump fails to handle WAL file with multi-page XLP_FIRST_IS_CONTRECORD data

2016-03-28 Thread Andres Freund
On 2016-03-28 13:21:41 +0530, Pavan Deolasee wrote: > On Wed, Mar 23, 2016 at 6:16 PM, Michael Paquier > wrote: > > > > > > > I'd just add dots at the end of the sentences in the comment blocks > > because that's project style, but I'm being picky, except that the > > logic looks quite good. > >

Re: [HACKERS] Move PinBuffer and UnpinBuffer to atomics

2016-03-28 Thread Andres Freund
M_IO_IN_PROGRESS; > - > - UnlockBufHdr(buf); > + state |= BM_IO_IN_PROGRESS; > + state &= ~BM_LOCKED; > + pg_atomic_write_u32(&buf->state, state); How about making UnlockBufHdr() take a new state parameter, and internally unset BM_LOCKED? > /* > + * Lock

Re: [HACKERS] backup tools ought to ensure created backups are durable

2016-03-29 Thread Andres Freund
On 2016-03-29 10:06:20 +0200, Magnus Hagander wrote: > On Tue, Mar 29, 2016 at 8:46 AM, Jim Nasby wrote: > > > On 3/28/16 11:03 AM, Magnus Hagander wrote: > > > >> > >> That should work yeah. And given that we already use that check in other > >> places, it seems it should be perfectly safe. And

Re: [HACKERS] [PROPOSAL] Client Log Output Filtering

2016-03-29 Thread Andres Freund
On 2016-03-29 12:28:40 -0400, Tom Lane wrote: > Alvaro Herrera writes: > > David Steele wrote: > >> On 3/29/16 10:18 AM, Tom Lane wrote: > >>> Repurposing COMMERROR is definitely starting to seem like a low-impact > >>> solution compared to these others. Under what circumstances would you > >>> b

Re: [HACKERS] [PROPOSAL] Client Log Output Filtering

2016-03-29 Thread Andres Freund
On 2016-03-29 18:33:19 +0200, Andres Freund wrote: > A couple years back I proposed making thinks like COMERROR into flags | > ed into elevel, rather than distinct levels. I still think that's a > better approach; and it doesn't force us to forgo using distinct

Re: [HACKERS] [PROPOSAL] Client Log Output Filtering

2016-03-29 Thread Andres Freund
On 2016-03-29 12:38:48 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2016-03-29 12:28:40 -0400, Tom Lane wrote: > If we invent LOG_ONLY (feel free to bikeshed the name), we could later > redefine it as (LOG | ERR_HIDE_FROM_CLIENT), if we ever upgrade the > underlying

Re: [HACKERS] [PROPOSAL] Client Log Output Filtering

2016-03-29 Thread Andres Freund
sn't look like he thought about the elevel-comparisons > issue either. Nope, I didn't. That's a long while ago ;) Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Move PinBuffer and UnpinBuffer to atomics

2016-03-29 Thread Andres Freund
On 2016-03-29 13:09:05 -0400, Robert Haas wrote: > On Mon, Mar 28, 2016 at 9:09 AM, Andres Freund wrote: > > On 2016-03-28 11:48:46 +0530, Dilip Kumar wrote: > >> On Sun, Mar 27, 2016 at 5:48 PM, Andres Freund wrote: > >> > What's sizeof(BufferDesc) after apply

Re: [HACKERS] Move PinBuffer and UnpinBuffer to atomics

2016-03-29 Thread Andres Freund
On 2016-03-29 13:24:40 -0400, Tom Lane wrote: > Andres Freund writes: > > Dilip, could you test performance of reducing ppc's spinlock to 1 byte? > > Cross-compiling suggest that doing so "just works". I.e. replace the > > #if defined(__ppc__) typedef from

Re: [HACKERS] Move PinBuffer and UnpinBuffer to atomics

2016-03-29 Thread Andres Freund
e LSN if we wrote a backup block. We aren't supposed > * to set this when only holding a share lock but as long as we > * serialise it somehow we're OK. We choose to set LSN while > * holding the buffer header lock, which causes any reader of an > * LSN who holds only a s

Re: [HACKERS] Move PinBuffer and UnpinBuffer to atomics

2016-03-29 Thread Andres Freund
On 2016-03-29 14:09:42 -0400, Tom Lane wrote: > Andres Freund writes: > > There's actually lbarx/stbcx - but it's not present in all ISAs. So I > > guess it's clear where to go. > > Hm. We could certainly add a configure test to see if the local assembler >

Re: [HACKERS] Move PinBuffer and UnpinBuffer to atomics

2016-03-30 Thread Andres Freund
On 2016-03-30 07:13:16 +0530, Dilip Kumar wrote: > On Tue, Mar 29, 2016 at 10:43 PM, Andres Freund wrote: > > > My gut feeling is that we should do both 1) and 2). > > > > Dilip, could you test performance of reducing ppc's spinlock to 1 byte? > > Cross-co

Re: [HACKERS] large regression for parallel COPY

2016-03-30 Thread Andres Freund
On 2016-03-30 15:50:21 -0400, Robert Haas wrote: > On Thu, Mar 10, 2016 at 8:29 PM, Andres Freund wrote: > > Allow to trigger kernel writeback after a configurable number of writes. > > While testing out Dilip Kumar's relation extension patch today, I > discovered (wit

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

2016-03-30 Thread Andres Freund
On 2016-03-28 22:50:49 +0530, Amit Kapila wrote: > On Fri, Sep 11, 2015 at 8:01 PM, Amit Kapila > wrote: > > > > On Thu, Sep 3, 2015 at 5:11 PM, Andres Freund wrote: > > > > > > > Updated comments and the patch (increate_clog_bufs_v2.patch) > > con

Re: [HACKERS] Question about DROP TABLE

2016-01-12 Thread Andres Freund
e corruption. If you disable autovacuum, you really should start vacuums in some other way. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Patch: fix lock contention for HASHHDR.mutex

2016-01-12 Thread Andres Freund
On 2016-01-11 21:16:42 -0300, Alvaro Herrera wrote: > Andres, Robert, are you still reviewing this patch? I don't think I've signed to - or actually did - reviewed this path. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To ma

Re: [HACKERS] checkpointer continuous flushing

2016-01-12 Thread Andres Freund
On 2016-01-12 17:50:36 +0530, Amit Kapila wrote: > On Tue, Jan 12, 2016 at 12:57 AM, Andres Freund wrote:> > > > > My theory is that this happens due to the sorting: pgbench is an update > > heavy workload, the first few pages are always going to be used if &

[HACKERS] Allow to specify (auto-)vacuum cost limits relative to the database/cluster size?

2016-01-12 Thread Andres Freund
Hi, right now the defaults for autovacuum cost limiting are so low that they regularly cause problems for our users. It's not exactly obvious that any installation above a couple gigabytes definitely needs to change autovacuum_vacuum_cost_delay & autovacuum_vacuum_cost_limit/vacuum_cost_limit. Esp

Re: [HACKERS] checkpointer continuous flushing

2016-01-12 Thread Andres Freund
have expected, the larger the database, the smaller > the impact of sorting & flushin on SSDs. Again: "HDD results are similar". I primarily tested on a 4 disk raid10 of 4 disks, and a raid0 of 20 disks. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] checkpointer continuous flushing

2016-01-12 Thread Andres Freund
this can lead to lesser merging of neighbouring > writes, but might not hurt if sync_file_range() API is cheap. The cost of writing out data doess correspond heavily with the number of random writes - which is what you get if you reduce the number of neighbouring writes. Greetings,

Re: [HACKERS] Speedup twophase transactions

2016-01-12 Thread Andres Freund
urs, which shouldn't be a problem > + * since the presence of long-lived prepared xacts indicates the > + * transaction manager isn't active. It's not *that* unlikely. Depending on settings the time between the computation of the redo pointer and CheckPointTwoPhase() isn't

Re: [HACKERS] Fwd: [JDBC] Re: 9.4-1207 behaves differently with server side prepared statements compared to 9.2-1102

2016-01-13 Thread Andres Freund
On January 13, 2016 3:02:27 PM GMT+01:00, Vladimir Sitnikov wrote: >> The custom plan is *more expensive*; > >You compare costs of custom vs generic plans. >I suggest: do not compare costs *at all*. > >>I don't know, it's your proposal :-) But it looks like I >misunderstood. > >It is not. > >My

Re: [HACKERS] Fwd: [JDBC] Re: 9.4-1207 behaves differently with server side prepared statements compared to 9.2-1102

2016-01-13 Thread Andres Freund
On 2016-01-13 17:26:43 +0300, Vladimir Sitnikov wrote: > > consider e.g a table with one somewhat common and otherwise just unique > > values.> > So what? > If I understand you properly, you mean: "if client sends unique binds > first 5-6 executions and bad non-unique afterwards, then cached plan

Re: [HACKERS] Fwd: [JDBC] Re: 9.4-1207 behaves differently with server side prepared statements compared to 9.2-1102

2016-01-13 Thread Andres Freund
ing > offset 0, offset 0*1, offset 0*2, etc kind of stuff they typically use > to tune queries) to convince plpgsql to use different statement ids. Basically you're arguing to fix one specific edge case which bugs you personally, by creating a lot of others, which don't bug you.

Re: [HACKERS] Performance degradation in commit ac1d794

2016-01-14 Thread Andres Freund
On 2015-12-26 12:22:48 +0100, Andres Freund wrote: > On 2015-12-25 16:29:53 -0500, Tom Lane wrote: > > Andres Freund writes: > > > There's a couple solutions I can think of to that problem: > > > 1) Use epoll()/kqueue, or other similar interfaces that don't r

Re: [HACKERS] Performance degradation in commit ac1d794

2016-01-14 Thread Andres Freund
ch more often? One way to address that would be to create a 'latch wait' datastructure, that'd then contain the epoll fd/win32 wait events/... That way you could have one 'LatchWait' for latch + client socket and one for latch + fdw sockets. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Performance degradation in commit ac1d794

2016-01-14 Thread Andres Freund
On January 14, 2016 5:16:59 PM GMT+01:00, Robert Haas wrote: >Yeah. Although I think for now it would be fine to just error out if >somebody tries to add a socket and there already is one. Then we >could lift that limitation in a later commit. Of course if Andres >wants to do the whole thing n

Re: [HACKERS] Performance degradation in commit ac1d794

2016-01-14 Thread Andres Freund
On 2016-01-14 11:31:03 -0500, Robert Haas wrote: > On Thu, Jan 14, 2016 at 10:56 AM, Andres Freund wrote: > I think your idea of a data structure the encapsulates a set of events > for which to wait is probably a good one. WaitLatch doesn't seem like > a great name. Maybe WaitE

Re: [HACKERS] Performance degradation in commit ac1d794

2016-01-14 Thread Andres Freund
On 2016-01-14 12:07:23 -0500, Robert Haas wrote: > > Do we want to provide a backward compatible API for all this? I'm fine > > either way. > > How would that work? I'm thinking of something like; int WaitOnLatchSet(LatchEventSet *set, int wakeEvents, long timeout); int WaitLatchOrSocket(volatil

Re: [HACKERS] Performance degradation in commit ac1d794

2016-01-14 Thread Andres Freund
On 2016-01-14 18:14:21 +0100, Andres Freund wrote: > I'm thinking of something like; > > int WaitOnLatchSet(LatchEventSet *set, int wakeEvents, long timeout); > > int > WaitLatchOrSocket(volatile Latch *latch, int wakeEvents, pgsocket sock,long > timeout) >

Re: [HACKERS] Performance degradation in commit ac1d794

2016-01-14 Thread Andres Freund
On 2016-01-14 18:14:21 +0100, Andres Freund wrote: > On 2016-01-14 12:07:23 -0500, Robert Haas wrote: > > > Do we want to provide a backward compatible API for all this? I'm fine > > > either way. > > > > How would that work? > > I'm th

Re: [HACKERS] Insert values() per-statement overhead

2016-01-15 Thread Andres Freund
On 2016-01-15 13:17:12 +0300, Vladimir Sitnikov wrote: > There is a finding that insert(x) values(y);insert(x) values(z); is > 2-4 times slower than insert(..) values(y),(z); > see [1], [2]. If you indeed just mean statements like above, without begin/commit, a large portion of the overhead will

Re: [HACKERS] checkpointer continuous flushing

2016-01-15 Thread Andres Freund
Hi Fabien, On 2016-01-11 14:45:16 +0100, Andres Freund wrote: > I measured it in a different number of cases, both on SSDs and spinning > rust. I just reproduced it with: > > postgres-ckpt14 \ > -D /srv/temp/pgdev-dev-800/ \ > -c maintenance_work_mem=2GB \ >

Re: [HACKERS] WIP: Rework access method interface

2016-01-16 Thread Andres Freund
On January 16, 2016 6:32:47 PM GMT+01:00, Tom Lane wrote: >As for modularity, nobody's moaned particularly about the >amcostestimate >functions all being in selfuncs.c. It all depends on what you think is >"modular". Well, back then you couldn't really have a production grade indeed as an exte

Re: [HACKERS] exposing pg_controldata and pg_config as functions

2016-01-16 Thread Andres Freund
On January 17, 2016 12:46:36 AM GMT+01:00, Michael Paquier wrote: , but we surely do not want to give away >checkpoint and recovery information. Why is that? A lot of that information is available anyway? --- Please excuse brevity and formatting - I am writing this on my mobile phone. -- Se

Re: [HACKERS] exposing pg_controldata and pg_config as functions

2016-01-18 Thread Andres Freund
ers are actually going to revoke the access > of those functions to public if we are going to make them > world-visible. It is easier as well to restrict things first, and then > relax if necessary, than the opposite as well. Meh, that seems pretty far into pseudo security arguments. Gr

Re: [HACKERS] Trivial fixes for some IDENTIFICATION comment lines

2016-01-18 Thread Andres Freund
On 2016-01-18 12:52:06 +0100, Shulgin, Oleksandr wrote: > I've noticed that in src/backend/replication/logical/reorderbuffer.c, the > IDENTIFICATION comment line is incorrect: > > * IDENTIFICATION > - * src/backend/catalog/dropcmds.c > - * src/backend/replication/logicalfuncs.c > - *

Re: [HACKERS] Re: BUG #13685: Archiving while idle every archive_timeout with wal_level hot_standby

2016-01-18 Thread Andres Freund
t record inserted at', and a way to not set that during insertion. Then during a checkpoint or the relevant bgwriter code, we look wether anything relevant happened in any backend since the last time we performed a checkpoint/logged a running xacts snapshot. Greetings, Andres Freund -- Se

Re: [HACKERS] checkpointer continuous flushing

2016-01-18 Thread Andres Freund
On 2016-01-16 10:01:25 +0100, Fabien COELHO wrote: > > Hello Andres, > > >I measured it in a different number of cases, both on SSDs and spinning > >rust. I just reproduced it with: > > > >postgres-ckpt14 \ > > -D /srv/temp/pgdev-dev-800/ \ > > -c maintenance_work_mem=2GB \ > >

Re: [HACKERS] [PATCH] Improve spinlock inline assembly for x86.

2016-01-18 Thread Andres Freund
On January 18, 2016 7:27:59 PM GMT+01:00, Robert Haas wrote: >On Sun, Jan 17, 2016 at 6:38 AM, Andreas Seltenreich > wrote: >> I'm currently experimenting with just-in-time compilation using >libfirm. >> While discussing issues with its developers, it was pointed out to me >> that our spinlock in

Re: [HACKERS] [PATCH] Improve spinlock inline assembly for x86.

2016-01-18 Thread Andres Freund
On January 18, 2016 10:42:42 PM GMT+01:00, Kevin Grittner wrote: >On Mon, Jan 18, 2016 at 3:04 PM, Andres Freund >wrote: >> On January 18, 2016 7:27:59 PM GMT+01:00, Robert Haas > wrote: >>> On Sun, Jan 17, 2016 at 6:38 AM, Andreas Seltenreich > wrote: > >>

Re: [HACKERS] exposing pg_controldata and pg_config as functions

2016-01-18 Thread Andres Freund
On January 18, 2016 11:10:35 PM GMT+01:00, Stephen Frost wrote: >* Robert Haas (robertmh...@gmail.com) wrote: >> On Mon, Jan 18, 2016 at 4:43 AM, Andres Freund >wrote: >> > Meh, that seems pretty far into pseudo security arguments. >> >> Yeah, I really don

Re: [HACKERS] [PATCH] Improve spinlock inline assembly for x86.

2016-01-18 Thread Andres Freund
On 2016-01-18 16:14:05 -0600, Kevin Grittner wrote: > Unconvinced that we should do performance testing on a proposed > performance patch before accepting it I'm unconvinced that it makes sense to view this as a performance patch. And unconvinced that you can sanely measure it. The lock prefix is

Re: [HACKERS] [PATCH] Improve spinlock inline assembly for x86.

2016-01-18 Thread Andres Freund
On 2016-01-18 16:56:22 -0600, Kevin Grittner wrote: > On Mon, Jan 18, 2016 at 4:50 PM, Andres Freund wrote: > > > Now I'm equally unconvinced that it's worthwhile to do anything > > here. I just don't think benchmarking plays a role either way. > > Well, t

Re: [HACKERS] checkpointer continuous flushing

2016-01-19 Thread Andres Freund
On 2016-01-19 10:27:31 +0100, Fabien COELHO wrote: > Also, the performance level is around 160 tps on HDDs, which make sense to > me for a 7200 rpm HDD capable of about x00 random writes per second. It > seems to me that you reported much better performance on HDD, but I cannot > really see how thi

Re: [HACKERS] checkpointer continuous flushing

2016-01-19 Thread Andres Freund
On 2016-01-19 13:34:14 +0100, Fabien COELHO wrote: > > >synchronous_commit = off does make a significant difference. > > Sure, but I had thought about that and kept this one... But why are you then saying this is fundamentally limited to 160 xacts/sec? > I think I found one possible culprit: I

Re: [HACKERS] checkpointer continuous flushing

2016-01-19 Thread Andres Freund
On 2016-01-19 12:58:38 -0500, Robert Haas wrote: > This seems like a problem with the WAL writer quite independent of > anything else. It seems likely to be inadvertent fallout from this > patch: > > Author: Simon Riggs > Branch: master Release: REL9_2_BR [4de82f7d7] 2011-11-13 09:00:57 + >

Re: [HACKERS] Logical decoding on standby

2016-01-19 Thread Andres Freund
only.) ephemeral slots are a different thing. Anyway, at this point failover slots aren't really proposed / have an agreed upon design yet, so it's a bit hard to take them into account. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] checkpointer continuous flushing

2016-01-20 Thread Andres Freund
On 2016-01-19 22:43:21 +0100, Andres Freund wrote: > On 2016-01-19 12:58:38 -0500, Robert Haas wrote: > > This seems like a problem with the WAL writer quite independent of > > anything else. It seems likely to be inadvertent fallout from this > > patch: > > > >

Re: [HACKERS] checkpointer continuous flushing

2016-01-20 Thread Andres Freund
On 2016-01-20 11:13:26 +0100, Andres Freund wrote: > On 2016-01-19 22:43:21 +0100, Andres Freund wrote: > > On 2016-01-19 12:58:38 -0500, Robert Haas wrote: > > I think the problem isn't really that it's flushing too much WAL in > > total, it's that it's

Re: [HACKERS] checkpointer continuous flushing

2016-01-20 Thread Andres Freund
On 2016-01-20 12:16:24 -0300, Alvaro Herrera wrote: > Andres Freund wrote: > > > The relevant thread is at > > http://archives.postgresql.org/message-id/CA%2BTgmoaCr3kDPafK5ygYDA9mF9zhObGp_13q0XwkEWsScw6h%3Dw%40mail.gmail.com > > what I didn't remember is that I v

Re: [HACKERS] Releasing in September

2016-01-20 Thread Andres Freund
On 2016-01-20 10:40:14 -0500, Bruce Momjian wrote: > We have gotten off of that cycle in the last two major releases, and > this isn't going to improve as long as we have commitfests starting > after January. I think this has very little to do with commitfest schedules, and much more with the "ear

Re: [HACKERS] Releasing in September

2016-01-20 Thread Andres Freund
On 2016-01-20 10:55:07 -0500, Robert Haas wrote: > It's certainly true that we twiddled our thumbs quite a bit about > getting 9.5 ready to ship. However, the old process where nobody > could get anything committed for six months out of the year blew > chunks, too. Personally, I think that the so

Re: [HACKERS] Releasing in September

2016-01-20 Thread Andres Freund
On 2016-01-20 11:19:28 -0500, Tom Lane wrote: > We will not get back to on-schedule releases unless we can keep -hackers > working on release testing/stabilization when it's time to do that, > rather than being distracted by shiny new stuff going into the next > release. Agreed. I'll note that the

Re: [HACKERS] Releasing in September

2016-01-20 Thread Andres Freund
On 2016-01-20 11:53:32 -0500, Robert Haas wrote: > But I'm not very sure that we're talking about the same set of people > here. If we're going to go to a system where nobody's allowed to > commit anything for the next release until the current release is > finalized, then we'd better have some pr

Re: [HACKERS] Releasing in September

2016-01-20 Thread Andres Freund
On 2016-01-20 09:15:01 -0800, Joshua D. Drake wrote: > On 01/20/2016 09:03 AM, Andres Freund wrote: > >If people don't fix the issues in time, there needs to be > >direct pushback, leading to much less stuff getting in next time round. > > > > We have been slowly

Re: [HACKERS] Releasing in September

2016-01-20 Thread Andres Freund
On 2016-01-20 12:18:28 -0500, Tom Lane wrote: > I'm not really sure why we've allowed CFs to drift on, though. Can't we > just arbitrarily decree them closed on the last day of the month? And > push unfinished work to the next one? Admittedly, this probably doesn't > work for the last CF of a re

<    1   2   3   4   5   6   7   8   9   10   >