Re: hot_standby_feedback vs excludeVacuum and snapshots

2018-06-11 Thread Simon Riggs
On 9 June 2018 at 15:41, Amit Kapila wrote: > On Fri, Jun 8, 2018 at 5:27 PM, Simon Riggs wrote: >> On 8 June 2018 at 11:33, Amit Kapila wrote: >>> On Fri, Jun 8, 2018 at 1:53 PM, Simon Riggs wrote: >>>> >>>> So the attached patch fixes both the bug in

Re: hot_standby_feedback vs excludeVacuum and snapshots

2018-06-09 Thread Simon Riggs
On 8 June 2018 at 19:03, Andres Freund wrote: > Hi, > > On 2018-06-08 09:23:02 +0100, Simon Riggs wrote: >> I have also found another bug which affects what we do next. >> >> For context, AEL locks are normally removed by COMMIT or ABORT. >> StandbyReleaseOldL

Re: hot_standby_feedback vs excludeVacuum and snapshots

2018-06-09 Thread Simon Riggs
On 8 June 2018 at 19:03, Andres Freund wrote: >> It seems to have affected Greg. > > As far as I can tell Greg was just theorizing? I'll wait for Greg to say whether this was an actual case that needs to be fixed or not. If not, happy to revert. -- Simon Riggs

Re: hot_standby_feedback vs excludeVacuum and snapshots

2018-06-08 Thread Simon Riggs
On 8 June 2018 at 11:33, Amit Kapila wrote: > On Fri, Jun 8, 2018 at 1:53 PM, Simon Riggs wrote: >> >> So the attached patch fixes both the bug in the recent commit and the >> one I just found by observation of 49bff5300d527, since they are >> related. >> >>

Re: hot_standby_feedback vs excludeVacuum and snapshots

2018-06-08 Thread Simon Riggs
scussion. Yes, it seems minor to me until it affects you, then its not. It seems to have affected Greg. The attached patch, or a later revision, needs to be backpatched to PG10 independently of the recent committed patch. I have yet to test this manually, but will do so tomorrow morning. -

Re: hot_standby_feedback vs excludeVacuum and snapshots

2018-06-07 Thread Simon Riggs
pecial case for this specific situation. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: computing completion tag is expensive for pgbench -S -M prepared

2018-06-07 Thread Simon Riggs
On 7 June 2018 at 20:27, Tom Lane wrote: > Simon Riggs writes: >> If we're going to compress the protocol, it seems sensible to remove >> extraneous information first. > > Breaking the wire protocol was nowhere in this thread. No, it wasn't. But there is a

Re: computing completion tag is expensive for pgbench -S -M prepared

2018-06-07 Thread Simon Riggs
On 7 June 2018 at 19:20, Andres Freund wrote: > On 2018-06-07 11:40:48 +0100, Simon Riggs wrote: >> On 7 June 2018 at 11:29, Pavel Stehule wrote: >> >> >> Do we actually need the completion tag at all? In most cases?? >> > >> > >> > af

Re: computing completion tag is expensive for pgbench -S -M prepared

2018-06-07 Thread Simon Riggs
On 7 June 2018 at 11:29, Pavel Stehule wrote: >> Do we actually need the completion tag at all? In most cases?? > > > affected rows is taken from this value on protocol level I didn't mean we should remove the number of rows. Many things rely on that. -- Simon Riggs

Re: computing completion tag is expensive for pgbench -S -M prepared

2018-06-07 Thread Simon Riggs
better Do we actually need the completion tag at all? In most cases?? Perhaps we should add a parameter to make it optional and turn it off by default, except for psql. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Possible bug in logical replication.

2018-06-07 Thread Simon Riggs
On 6 June 2018 at 17:22, Alvaro Herrera wrote: > This thread seems to have died down without any fix being proposed. > Simon, you own this open item. Thanks, will look. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Tr

Re: I'd like to discuss scaleout at PGCon

2018-06-06 Thread Simon Riggs
f where data is located (difficult to scale) 2. you must put data in all places the client can connect to (i.e. multimaster) Perhaps there are more? -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: plans for PostgreSQL 12

2018-06-03 Thread Simon Riggs
eresting or not. Sounds good. I think this would need to be restricted by operator and datatype, since in general you won't know if the datatype functions need a snapshot or not. Immutable functions for the operators ought to do it, but I think that might not be enough. -- Simon Riggs

Re: I'd like to discuss scaleout at PGCon

2018-06-02 Thread Simon Riggs
ture in-core scalablility solution. Each incoming query needs to be planned and executed from one coordinator component, then the work performed across many workers on different nodes (or just one). We could have coordinator components on each worker node, or we could have a set of coordinator nodes and a set of worker nodes. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: I'd like to discuss scaleout at PGCon

2018-06-02 Thread Simon Riggs
On 1 June 2018 at 16:56, Ashutosh Bapat wrote: > On Fri, Jun 1, 2018 at 11:10 AM, Simon Riggs wrote: >> >> Using a central coordinator also allows multi-node transaction >> control, global deadlock detection etc.. > > But that becomes an SPOF and then we have to config

Re: I'd like to discuss scaleout at PGCon

2018-06-01 Thread Simon Riggs
. For example, supporting SQL query plans that allow for redistribution of data for joins. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: I'd like to discuss scaleout at PGCon

2018-06-01 Thread Simon Riggs
rsing, we might want to find a way to pass > a "normalized" parse tree down to the foreign server. We need to > normalize the OIDs in the parse tree since those may be different > across the nodes. Passing detailed info between servers is exactly what XL does. It requires

Re: SHOW ALL does not honor pg_read_all_settings membership

2018-05-31 Thread Simon Riggs
atch to v10? > GetConfigOption is wrong by the way, as restrict_superuser means that > all members of the group pg_read_all_settings can read > GUC_SUPERUSER_ONLY params, and not only superusers, so the comment at > least needs a fix, the variable ought to be renamed as well. OK --

Re: pg_replication_slot_advance to return NULL instead of 0/0 if slot not advanced

2018-05-31 Thread Simon Riggs
oblem here is there are no comments explaining how to access the various fields in the structure, so there was no way to check whether the code was good or not. If we add corrective code we should clarify that in comments the .h file also, as is done in XlogCtl Your points look correct to me, w

Re: Undo logs

2018-05-28 Thread Simon Riggs
that need to be discussed, so lets crack on and discuss them please. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: pg_replication_slot_advance to return NULL instead of 0/0 if slot not advanced

2018-05-25 Thread Simon Riggs
we try to move it backwards, but if that one fails we return > InvalidXLogRecPtr (which can happen in the case of concurrent activity on > the slot, I think)? In this case, maybe we should just re-check that and > raise an error appropriately? Agreed > (I haven't looked at the logical slot path, but I assume it would have > something similar in it) -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: PG 11 feature count

2018-05-18 Thread Simon Riggs
220 > 10 194 > 11 167 It would be useful to combine that with the CF app data showing number of patches submitted and number of features rejected. Not available for all time, but certainly goes back a few years now. -- Simon Riggshttp://www.2ndQuadr

Re: Removing unneeded self joins

2018-05-16 Thread Simon Riggs
On 16 May 2018 at 15:10, Tom Lane wrote: > Simon Riggs writes: >> What I would add is that I've seen cases where the extra joins do NOT >> hurt performance, so the extra CPU used to remove the join hurts more >> than the benefit of removing it. Yes, we tried it. >

Re: Removing unneeded self joins

2018-05-16 Thread Simon Riggs
t; like to see us do something about it. I wish we could optimize away > inner joins, too, for similar reasons. I agree with everything you say. What I would add is that I've seen cases where the extra joins do NOT hurt performance, so the extra CPU used to remove the join hurts more than th

Re: [HACKERS] Surjective functional indexes

2018-05-11 Thread Simon Riggs
On 11 May 2018 at 16:37, Andres Freund wrote: > On 2018-05-11 14:56:12 +0200, Simon Riggs wrote: >> On 11 May 2018 at 05:32, Andres Freund wrote: >> > No. Simon just claimed it's not actually a concern: >> > https://www.postgresql.org/message-id/canp8+j+vtskphep

Re: [HACKERS] Surjective functional indexes

2018-05-11 Thread Simon Riggs
h significantly in response to those reviews, as well as explaining why the patch is fine as it is. It's a useful patch that PostgreSQL needs, so all good. I have no problem if you want to replace this with an even better design in a later release. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Indexes on partitioned tables and foreign partitions

2018-05-11 Thread Simon Riggs
On 9 May 2018 at 17:33, Robert Haas wrote: > On Wed, May 9, 2018 at 11:20 AM, Simon Riggs wrote: >> On 9 May 2018 at 16:15, Robert Haas wrote: >>> On Wed, May 9, 2018 at 11:14 AM, Simon Riggs wrote: >>>> On 9 May 2018 at 16:10, Tom Lane wrote: >>>>&

Re: Needless additional partition check in INSERT?

2018-05-09 Thread Simon Riggs
se of this oversight is the lack of comments to explain the original coding, so we need to correct that in this patch, please. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Indexes on partitioned tables and foreign partitions

2018-05-09 Thread Simon Riggs
On 9 May 2018 at 16:15, Robert Haas wrote: > On Wed, May 9, 2018 at 11:14 AM, Simon Riggs wrote: >> On 9 May 2018 at 16:10, Tom Lane wrote: >>> Robert Haas writes: >>>> On Wed, May 9, 2018 at 9:08 AM, Simon Riggs wrote: >>>>> Shouldn't the fix

Re: Indexes on partitioned tables and foreign partitions

2018-05-09 Thread Simon Riggs
On 9 May 2018 at 16:10, Tom Lane wrote: > Robert Haas writes: >> On Wed, May 9, 2018 at 9:08 AM, Simon Riggs wrote: >>> Shouldn't the fix be to allow creation of indexes on foreign tables? >>> (Maybe they would be virtual or foreign indexes??) > >> It m

Re: Indexes on partitioned tables and foreign partitions

2018-05-09 Thread Simon Riggs
a foreign table, why can we not just assume a unique index exists?? Why the difference? -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Indexes on partitioned tables and foreign partitions

2018-05-09 Thread Simon Riggs
On 9 May 2018 at 15:26, Arseny Sher wrote: > > Simon Riggs writes: > >> How much sense is it to have a partitioned table with a mix of local >> and foreign tables? > > Well, as much sense as fdw-based sharding has, for instance. It is > arguable, but it exists. >

Re: Indexes on partitioned tables and foreign partitions

2018-05-09 Thread Simon Riggs
on of indexes on foreign tables? (Maybe they would be virtual or foreign indexes??) -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Postgres, fsync, and OSs (specifically linux)

2018-04-29 Thread Simon Riggs
On 28 April 2018 at 09:15, Andres Freund wrote: > Hi, > > On 2018-04-28 08:25:53 -0700, Simon Riggs wrote: >> > - Use direct IO. Due to architectural performance issues in PG and the >> > fact that it'd not be applicable for all installations I don't thin

Re: Postgres, fsync, and OSs (specifically linux)

2018-04-29 Thread Simon Riggs
On 28 April 2018 at 08:25, Simon Riggs wrote: > On 27 April 2018 at 15:28, Andres Freund wrote: > >> - Add a pre-checkpoint hook that checks for filesystem errors *after* >> fsyncing all the files, but *before* logging the checkpoint completion >> record. Operating s

Re: Postgres, fsync, and OSs (specifically linux)

2018-04-28 Thread Simon Riggs
dmission of defeat for us to take that as our main route to a solution. The people I've spoken to so far have encouraged us to continue working with the filesystem layer, offering encouragement of our decision to use filesystems. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Query is over 2x slower with jit=on

2018-04-18 Thread Simon Riggs
ns before this gets too far into the wild. SSL is somewhat understandable because its not a Postgres-private term. geqo is regrettable and we really don't want any more too short/abbreviated parameter names. Think of our EOU if every GUC was a TLA. Thanks -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: reloption to prevent VACUUM from truncating empty pages at the end of relation

2018-04-18 Thread Simon Riggs
lock itself can be cancelled, so the user would also be interested in explicitly requesting a retry with a separate command/function. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Bugs in TOAST handling, OID assignment and redo recovery

2018-04-11 Thread Simon Riggs
On 11 April 2018 at 19:57, Tom Lane wrote: > Pavan Deolasee writes: >> Ok. I propose attached patches, more polished this time. > > I'll take these, unless some other committer is hot to do so? Please go ahead. -- Simon Riggshttp://www.2ndQuadrant.com/ Po

Re: pgsql: New files for MERGE

2018-04-11 Thread Simon Riggs
On 7 April 2018 at 18:45, Tom Lane wrote: > Simon Riggs writes: >> On 6 April 2018 at 17:22, Bruce Momjian wrote: >>> My point was that people didn't ask you to work harder on fixing the >>> patch, but in reverting it. You can work harder on fixing things i

Re: pgsql: New files for MERGE

2018-04-07 Thread Simon Riggs
On 6 April 2018 at 17:22, Bruce Momjian wrote: > On Fri, Apr 6, 2018 at 09:21:54AM +0100, Simon Riggs wrote: >> On 5 April 2018 at 21:02, Bruce Momjian wrote: >> > Simon, you have three committers in this thread suggesting this patch be >> > reverted. Are you just

Re: Add support for printing/reading MergeAction nodes

2018-04-06 Thread Simon Riggs
On 4 April 2018 at 18:08, Tom Lane wrote: > Simon Riggs writes: >> On 4 April 2018 at 17:19, Tom Lane wrote: >>> BTW, poking around in the grammar, I notice that MergeStmt did not >>> get added to RuleActionStmt. That seems like a rather serious >>> omiss

Re: [HACKERS] MERGE SQL Statement for PG11

2018-04-06 Thread Simon Riggs
think that people could spend their time better - and issue that concerns me also. But that certainly doesn't apply to parts of the code like this where we have full test coverage. It may not even apply to recovery now we have the ability to check in real-time the results of recovery and

Re: pgsql: New files for MERGE

2018-04-06 Thread Simon Riggs
On 5 April 2018 at 21:02, Bruce Momjian wrote: > On Thu, Apr 5, 2018 at 11:15:20AM +0100, Simon Riggs wrote: >> On 4 April 2018 at 21:28, Simon Riggs wrote: >> > On 4 April 2018 at 21:14, Andres Freund wrote: >> > >> >>> The normal way is to ma

Re: [HACKERS] MERGE SQL Statement for PG11

2018-04-05 Thread Simon Riggs
On 5 April 2018 at 17:07, Alvaro Herrera wrote: > Simon Riggs wrote: >> On 5 April 2018 at 16:09, Alvaro Herrera wrote: >> > Quick item: parse_clause.h fails cpluspluscheck because it has a C++ >> > keyword as a function argument name: >> > >> > ./s

Re: [HACKERS] MERGE SQL Statement for PG11

2018-04-05 Thread Simon Riggs
List **namespace); > ^ How's this as a fix? -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services fnamespace.v1.patch Description: Binary data

Re: Excessive PostmasterIsAlive calls slow down WAL redo

2018-04-05 Thread Simon Riggs
is would only > affect the case where we're actively replaying records. +1 -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] MERGE SQL Statement for PG11

2018-04-05 Thread Simon Riggs
.co.jp > > and his suggestion to use special macro looks better for me: > - charrelkind; > + charrelkind PG_USED_FOR_ASSERTS_ONLY; Thanks both, I already fixed that. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] MERGE SQL Statement for PG11

2018-04-05 Thread Simon Riggs
On 5 April 2018 at 13:19, Jesper Pedersen wrote: > Hi, > > On 04/05/2018 08:04 AM, Simon Riggs wrote: >> >> On 5 April 2018 at 12:56, Jesper Pedersen >> wrote: >>> >>> Updated for non-assert build. >> >> >> Thanks, pushed. Sorry to h

Re: [HACKERS] MERGE SQL Statement for PG11

2018-04-05 Thread Simon Riggs
On 5 April 2018 at 12:56, Jesper Pedersen wrote: > Hi, > > On 04/05/2018 07:48 AM, Simon Riggs wrote: >>> >>> Updated version due to latest refactoring. >> >> >> Thanks for your input. Removing that seems to prevent compilation. >> >> Did s

Re: [HACKERS] MERGE SQL Statement for PG11

2018-04-05 Thread Simon Riggs
On 5 April 2018 at 12:38, Jesper Pedersen wrote: > Hi Simon and Paven, > > On 04/04/2018 08:46 AM, Jesper Pedersen wrote: >> >> On 03/30/2018 07:10 AM, Simon Riggs wrote: >>> >>> No problems found, but moving proposed commit to 2 April pm >>>

Re: Add support for printing/reading MergeAction nodes

2018-04-05 Thread Simon Riggs
ags. No problem there too. OK, so $OP fixed. > This now also enforces single VALUES clause in the grammar itself instead of > doing that check at parse-analyse time. So that's a net improvement too. OK, that's good. I've updated the docs to show this restriction correctly.

Re: pgsql: New files for MERGE

2018-04-05 Thread Simon Riggs
On 4 April 2018 at 21:28, Simon Riggs wrote: > On 4 April 2018 at 21:14, Andres Freund wrote: > >>> The normal way is to make review comments that allow change. Your >>> request for change of the parser data structures is fine and can be >>> done, possibly by Sat

Re: [HACKERS] MERGE SQL Statement for PG11

2018-04-05 Thread Simon Riggs
e. (Now that the patch is committed, I don't know if there > would be a rethink about changing file names. May be not, just raising that > concern) My review notes suggest a file called execMerge.c. I didn't spot the filename change. I think it's important to do that because there is no executor node called Merge. That is especially confusing because there *is* an executor node called MergeAppend and we want some cognitive distance between those things. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: pgsql: New files for MERGE

2018-04-04 Thread Simon Riggs
gnored those requests. Pavan tells me he has replied to you and is working on specific changes. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: pgsql: New files for MERGE

2018-04-04 Thread Simon Riggs
. There has been no discussion of what exactly would be better, only that what we have is somehow wrong, a point which both Pavan and I dispute, not least because the executor has already been rewritten once at Peter's request. I was under no pressure at all to commit this. In my opinion this

Re: Add support for printing/reading MergeAction nodes

2018-04-04 Thread Simon Riggs
On 4 April 2018 at 18:51, Tom Lane wrote: > Simon Riggs writes: >> On 4 April 2018 at 17:19, Tom Lane wrote: >>> If the MERGE patch has broken this, I'm going to push back on that >>> and push back on it hard, because it probably means there are a >>>

Re: Add support for printing/reading MergeAction nodes

2018-04-04 Thread Simon Riggs
d Delete and these could be output in various debug modes. I'm not clear what meaning we might attach to them if we looked since that differs from normal INSERTs, UPDATEs, DELETEs, but lets see. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Add support for printing/reading MergeAction nodes

2018-04-04 Thread Simon Riggs
mt? Could you explain what command you think should be supported? -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [PATCH] Logical decoding of TRUNCATE

2018-04-04 Thread Simon Riggs
s reasonable to treat it > specially. Real DDL is being worked on in the 2PC decoding thread. > What we are discussing here isn't going to be applicable there and vice > versa, I think. In fact, one of the reasons for this effort is that in > BDR TRUNCATE is currently handled like

Re: [PATCH] Logical decoding of TRUNCATE

2018-04-02 Thread Simon Riggs
>>> >>> >> >> Why is this a good idea? > > I think it seemed like a good idea at the time, so to speak, but several > people have argued against it, so we should probably change this in the > final version. Who has argued aga

Re: [HACKERS] logical decoding of two-phase transactions

2018-04-02 Thread Simon Riggs
; which would indicate this fact - logical decoding is using it's own > snapshots it could inject the information about being inside the 2PC > decoding. Perhaps, but how do we know we've covered all the right places? We don't know what every plugin will require, do we? The pl

Re: [HACKERS] logical decoding of two-phase transactions

2018-04-02 Thread Simon Riggs
r make an unknown task happen automatically? We can't. We have a reasonable approach here. Locking shared resources before using them is not a radical new approach, its just standard development. If we find a better way in the future, we can use that, but requiring a better solution wh

Re: [PATCH] Logical decoding of TRUNCATE

2018-04-01 Thread Simon Riggs
e logically >> decoded. >> + * We could optimize this away when !RelationIsLogicallyLogged(rel) >> + * but that doesn't save much space or time. > > What you're saying isn't that you're not logging anything, but that > you're allocating the

Re: hot_standby_feedback vs excludeVacuum and snapshots

2018-04-01 Thread Simon Riggs
VACUUM FULL was the thing that needed to be excluded in the past because it needed an xid to move rows. Greg's concern is a good one and his noticing that we hadn't specifically excluded VACUUMs is valid, so we should exclude them. Well spotted, Greg. So although this doesn't have t

Re: [HACKERS] MERGE SQL Statement for PG11

2018-03-30 Thread Simon Riggs
On 29 March 2018 at 10:50, Simon Riggs wrote: > On 28 March 2018 at 12:00, Pavan Deolasee wrote: > >> v27 attached, though review changes are in >> the add-on 0005 patch. > > This all looks good now, thanks for making all of those changes. > > I propose [v27 patch1

Re: Changing WAL Header to reduce contention during ReserveXLogInsertLocation()

2018-03-30 Thread Simon Riggs
gain the observation that they are no longer used for replication and if they are used as well as replication, have a bad effect on performance. I think it would be easily possible to add some more detail to the WAL stream if needed. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Changing WAL Header to reduce contention during ReserveXLogInsertLocation()

2018-03-29 Thread Simon Riggs
in fact >> strong enough that it's good enough for government work. He might be > > Is he? I think the claims in this thread were pretty much that xl_curr > and xl_prev provide the same level of protection. Yes, the blockchain analogy breaks down because we don't include p

Re: Changing WAL Header to reduce contention during ReserveXLogInsertLocation()

2018-03-29 Thread Simon Riggs
/boundaries) >> >> That seems very much like xl_prev. > > I don't think so, because this ignores, for example, timeline > switches, or multiple clusters accidentally sharing an archive > directory. I'm not hearing any actual technical problems. > Given where we are

Re: Changing WAL Header to reduce contention during ReserveXLogInsertLocation()

2018-03-29 Thread Simon Riggs
On 29 March 2018 at 18:13, Tomas Vondra wrote: > On 03/29/2018 06:42 PM, Tom Lane wrote: >> Simon Riggs writes: >>> I know the approach is new and surprising but I thought about it a lot >>> before proposing it and I couldn't see any holes; still can't. Pl

Re: Changing WAL Header to reduce contention during ReserveXLogInsertLocation()

2018-03-29 Thread Simon Riggs
e. I know the approach is new and surprising but I thought about it a lot before proposing it and I couldn't see any holes; still can't. Please give this some thought so we can get comfortable with this idea and increase performance as a result. Thanks. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] MERGE SQL Statement for PG11

2018-03-29 Thread Simon Riggs
ill no doubt discover fairly soon after commit, such as additional runs by SQLsmith and more eyeballs. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] A design for amcheck heapam verification

2018-03-29 Thread Simon Riggs
CREATE INDEX afterwards anyway, which makes it more likely that people would just run CREATE INDEX and not bother with the check. So my initial questions are about when we would run this and making sure that is documented. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Commit fest 2017-11

2018-03-29 Thread Simon Riggs
future of course, unless somebody wants to go > back and backfill existing patches with such information (which they might > be). The focus of this is on the Committers, which seems wrong. I suggest someone does another analysis that shows how many patch reviews have been conducted by patch authors, so we can highlight people who are causing the problem yet not helping solve the problem. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] MERGE SQL Statement for PG11

2018-03-28 Thread Simon Riggs
On 29 March 2018 at 07:37, Pavan Deolasee wrote: > > > On Tue, Mar 27, 2018 at 5:00 PM, Simon Riggs wrote: >> >> >> In terms of further performance optimization, if there is just one >> WHEN AND condition and no unconditional WHEN clauses then we can add >>

Re: [HACKERS] logical decoding of two-phase transactions

2018-03-28 Thread Simon Riggs
On 28 March 2018 at 16:28, Nikhil Sontakke wrote: > Simon, 0003-Add-GID-and-replica-origin-to-two-phase-commit-abort.patch > is the exact patch that you had posted for an earlier commit. 0003 Pushed -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 S

Re: [HACKERS] Surjective functional indexes

2018-03-27 Thread Simon Riggs
On 23 March 2018 at 15:54, Simon Riggs wrote: > So please could you make the change? Committed, but I still think that change would be good. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Changing WAL Header to reduce contention during ReserveXLogInsertLocation()

2018-03-27 Thread Simon Riggs
gt;> 2. Does the new logic in pg_rewind to search backward for a checkpoint >> work reliably, and will it be slow? > > If you have to search backwards, this breaks it. Full stop. You don't have to search backwards. We only need to locate the last checkpoint record. -- Simon Ri

Re: [HACKERS] MERGE SQL Statement for PG11

2018-03-27 Thread Simon Riggs
On 27 March 2018 at 11:46, Simon Riggs wrote: > On 27 March 2018 at 10:31, Pavan Deolasee wrote: > >> Fixed in v26. > > More comments on v26 In terms of further performance optimization, if there is just one WHEN AND condition and no unconditional WHEN clauses then we ca

Re: [HACKERS] MERGE SQL Statement for PG11

2018-03-27 Thread Simon Riggs
coverage of these ERROR messages Named security policy violation SELECT not allowed in MERGE INSERT... Multiple VALUES clauses not... MERGE is not supported for this... MERGE is not supported for relations with inheritance MERGE is not supported for relations with rules -- Simon Riggs

Re: [HACKERS] MERGE SQL Statement for PG11

2018-03-27 Thread Simon Riggs
de since the > mergrSourceTargetList only includes user columns and so set_plan_refs() > complains about a system column. > > I am not sure what's the best way to handle this. May be we can add system > columns to the mergrSourceTargetList. I haven't yet found a neat way to do > that. I was saying the comment needs changing, not the code. Cool, thanks -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] logical decoding of two-phase transactions

2018-03-27 Thread Simon Riggs
On 23 March 2018 at 15:26, Simon Riggs wrote: > Reviewing 0003-Add-support-for-logging-GID-in-commit-abort-WAL-reco > > Looks fine, reworked patch attached > * added changes to xact.h from patch 4 so that this is a whole, > committable patch > * added comments to make abort

Re: [HACKERS] MERGE SQL Statement for PG11

2018-03-27 Thread Simon Riggs
On 26 March 2018 at 17:06, Simon Riggs wrote: > On 26 March 2018 at 15:39, Pavan Deolasee wrote: > > That's all I can see so far. * change comment “once to” to “once” in src/include/nodes/execnodes.h * change comment “and to run” to “and once to run” * change “result relati

Re: [HACKERS] MERGE SQL Statement for PG11

2018-03-27 Thread Simon Riggs
On 26 March 2018 at 23:10, Peter Geoghegan wrote: > On Mon, Mar 26, 2018 at 12:17 PM, Simon Riggs wrote: >>> As far as I >>> know, the proposed MERGE patch has that issue an existing DML commands >>> don't; but someone else may have better information. >&

Re: [HACKERS] MERGE SQL Statement for PG11

2018-03-26 Thread Simon Riggs
On 26 March 2018 at 17:52, Robert Haas wrote: > On Mon, Mar 26, 2018 at 12:16 PM, Simon Riggs wrote: >> On 26 March 2018 at 16:09, Robert Haas wrote: >>> On Mon, Mar 26, 2018 at 5:53 AM, Simon Riggs wrote: >>>> Since we now have MVCC catalog scans, all the name lo

Re: [HACKERS] MERGE SQL Statement for PG11

2018-03-26 Thread Simon Riggs
On 26 March 2018 at 16:09, Robert Haas wrote: > On Mon, Mar 26, 2018 at 5:53 AM, Simon Riggs wrote: >> Since we now have MVCC catalog scans, all the name lookups are >> performed using the same snapshot so in the above scenario the newly >> created object would be invisib

Re: [HACKERS] MERGE SQL Statement for PG11

2018-03-26 Thread Simon Riggs
o system column is allowed" 11. In comment "Since the plan re-evaluated by EvalPlanQual uses the second RTE", suggest using "join RTE" to make it more explicit which RTE we are discussing 12. Missed out merge.sgml from v25 patch. 13. For triggers we say &qu

Re: [HACKERS] MERGE SQL Statement for PG11

2018-03-26 Thread Simon Riggs
v25 is the "commit candidate" and we can add other patches to it. Given recent bugfix/changes I don't plan to commit this tomorrow anymore. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] MERGE SQL Statement for PG11

2018-03-26 Thread Simon Riggs
rror if they aren't. Since we now have MVCC catalog scans, all the name lookups are performed using the same snapshot so in the above scenario the newly created object would be invisible to the second name lookup. So I don't see anyway for the ERROR to occur and hence no need for a cross check, for UPDATE or MERGE. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Changing WAL Header to reduce contention during ReserveXLogInsertLocation()

2018-03-24 Thread Simon Riggs
On 24 March 2018 at 12:19, Robert Haas wrote: > On Sat, Mar 24, 2018 at 8:11 AM, Simon Riggs wrote: >> On 24 March 2018 at 11:58, Robert Haas wrote: >>> On Sat, Mar 24, 2018 at 7:42 AM, Simon Riggs wrote: >>>> I suggest we focus on the engineering. I've not d

Re: [HACKERS] MERGE SQL Statement for PG11

2018-03-24 Thread Simon Riggs
against b.foo and the other half against a.foo. I don't > know whether that will crash or bomb out with a strange error or just > make some unexpected modification to one of those tables, but the > behavior, even if not insecure, will certainly be wrong. MERGE uses multiple RTEs in

Re: Changing WAL Header to reduce contention during ReserveXLogInsertLocation()

2018-03-24 Thread Simon Riggs
On 24 March 2018 at 11:58, Robert Haas wrote: > On Sat, Mar 24, 2018 at 7:42 AM, Simon Riggs wrote: >> I suggest we focus on the engineering. I've not discussed this patch >> with Pavan offline. > > Well then proposing to commit moments after it's been posted is &g

Re: Changing WAL Header to reduce contention during ReserveXLogInsertLocation()

2018-03-24 Thread Simon Riggs
I suggest we focus on the engineering. I've not discussed this patch with Pavan offline. On 23 March 2018 at 23:32, Michael Paquier wrote: > On Fri, Mar 23, 2018 at 11:06:48AM +0000, Simon Riggs wrote: >> Your assumption that I would commit a new patch that was 29 mins old >&g

Re: [HACKERS] Surjective functional indexes

2018-03-23 Thread Simon Riggs
tions it. Above that in RelationData we have other structures that are List of OIDs, so Alvaro's proposal make sense. That would simplify the code in ProjectionIsNotChanged() by just looping over the list of projection indexes rather than the list of indexes So please could you make the

Re: [HACKERS] logical decoding of two-phase transactions

2018-03-23 Thread Simon Riggs
patch 4 so that this is a whole, committable patch * added comments to make abort and commit structs look same Attached patch is proposed for a separate, early commit as part of this -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Trai

Re: [HACKERS] MERGE SQL Statement for PG11

2018-03-23 Thread Simon Riggs
tests, and so far it >>> holds up. Peter, if you have the code and you consider it important that this subfeature is in PostgreSQL, why not post the code so we can commit it? Why would we repeat what has already been done? -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Changing WAL Header to reduce contention during ReserveXLogInsertLocation()

2018-03-23 Thread Simon Riggs
On 23 March 2018 at 09:22, Michael Paquier wrote: > On Fri, Mar 23, 2018 at 09:04:55AM +0000, Simon Riggs wrote: >> So it shows clear benefit for both bulk actions and OLTP, with no >> regressions. >> >> No objection exists to the approach used in the patch, so I&#x

Re: Changing WAL Header to reduce contention during ReserveXLogInsertLocation()

2018-03-23 Thread Simon Riggs
On 23 March 2018 at 08:35, Pavan Deolasee wrote: > > > On Fri, Mar 9, 2018 at 8:49 PM, Peter Eisentraut > wrote: >> >> On 2/1/18 19:21, Simon Riggs wrote: >> > If we really can't persuade you of that, it doesn't sink the patch. We >> > can have

Re: Changing WAL Header to reduce contention during ReserveXLogInsertLocation()

2018-03-23 Thread Simon Riggs
On 2 February 2018 at 02:17, Michael Paquier wrote: > On Fri, Feb 02, 2018 at 12:21:49AM +0000, Simon Riggs wrote: >> Yes, it would be about 99% of the time. > > When it comes to recovery, I don't think that 99% is a guarantee > sufficient. (Wondering about the maths behin

Re: [HACKERS] Surjective functional indexes

2018-03-22 Thread Simon Riggs
een around the actual user interface to request it. Index option handling has changed (and this needs rebase!), but other than that I think we want this and am planning to commit something early next week. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

<    1   2   3   4   5   6   7   8   >