Re: [HACKERS] Lets (not) break all the things. Was: [pgsql-advocacy] 9.6 -> 10.0

2016-04-11 Thread Craig Ringer
> + * v4 protocol (feature negotiation, lazy blob fetching, etc) * retire pg_hba.conf and use SQL access management ? -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] WIP: Failover Slots

2016-04-08 Thread Craig Ringer
tex); was_dirty = slot->dirty; slot->just_dirtied = false; SpinLockRelease(>mutex); /* and don't do anything if there's nothing to write */ if (!was_dirty) return; } ... though I think what I really should've done there i

Re: [HACKERS] Timeline following for logical slots

2016-04-07 Thread Craig Ringer
agree on a > direction that we can all live with. Yeah. I'd like to see us able to work on a shared dev tree rather than mailing patches around, at least. I'm not going to make it to PgConf.US. I don't know about Ottawa yet, but I doubt it given that I did go to Brussels. Perth is absurdly far from almost everywhere, unfortunately. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Proposal: Generic WAL logical messages

2016-04-06 Thread Craig Ringer
. In some ways it seems like the argument to pg_logical_emit_message(...) should be 'bytea'. That'd be much more convenient for application use. But then it's a pain when using it via the text-format SQL interface calls, where we've got no sensible way to output it. -- Craig Ringer

Re: [HACKERS] WIP: Failover Slots

2016-04-06 Thread Craig Ringer
A few thoughts on failover slots vs the alternative of pushing catalog_xmin up to the master via a replica's slot and creating independent slots on replicas. Failover slots: --- + Failover slots are very easy for applications. They "just work" and are transparent for failover. This is great

Re: [HACKERS] WIP: Failover Slots

2016-04-06 Thread Craig Ringer
On 6 April 2016 at 17:43, Simon Riggs <si...@2ndquadrant.com> wrote: > On 25 January 2016 at 14:25, Craig Ringer <cr...@2ndquadrant.com> wrote: > > >> I'd like to get failover slots in place for 9.6 since the're fairly >> self-contained and meet an immediate need:

Re: [HACKERS] Timeline following for logical slots

2016-04-06 Thread Craig Ringer
Draft patch for comment fix.​ From 304453f67b26219dc94e6aef9dfdc1c1debf12bb Mon Sep 17 00:00:00 2001 From: Craig Ringer <cr...@2ndquadrant.com> Date: Wed, 6 Apr 2016 14:57:26 +0800 Subject: [PATCH] Fix incorrect comments introduced in logical decoding timeline following --- src/b

Re: [HACKERS] Timeline following for logical slots

2016-04-06 Thread Craig Ringer
I tried to address Andres's entirely valid complaints about that comment I added in this patch. I was going to add a description to restart_lsn in slot.h instead, but it's proving harder to accurately describe restart_lsn than I'd like. It's a point at which no transactions that commit after

Re: [HACKERS] pg_xlogdump fails to handle WAL file with multi-page XLP_FIRST_IS_CONTRECORD data

2016-04-05 Thread Craig Ringer
On 1 April 2016 at 21:30, Craig Ringer <cr...@2ndquadrant.com> wrote: > I'll attach the new testcase once I either get it to reproduce this bug or > give up and leave the basic xlogdump testcase alone. > I had another bash at this and I still can't reproduce it on master using t

Re: [HACKERS] Updated backup APIs for non-exclusive backups

2016-04-05 Thread Craig Ringer
spend several solid days reorganizing and updating those docs some day, but I think that for now moving some chunks around will still help. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] WIP: Failover Slots

2016-04-05 Thread Craig Ringer
hink it's valuable and want to run with it, but we're very close to feature freeze now. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Timeline following for logical slots

2016-04-05 Thread Craig Ringer
gt; step up. It's a big job, and there are lots of differing viewpoints on it. It does seem like having 2ndQ submit a patch, wait for feedback, be told "oh, don't do it that way, come back with ", then wait for us to respond and address the issues isn't working. Certainly isn't going to get us there in a timely manner. I want to explore some options here, but I have to talk to internal team members. Sorry for the handwavyness of that. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Timeline following for logical slots

2016-04-05 Thread Craig Ringer
ding replica when we get support for that. By contrast, this approach would actually help solve one of the problems needed to get replay from a replica working. It's a pity it's a no-hoper for 9.6 though. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Timeline following for logical slots

2016-04-05 Thread Craig Ringer
l logical rep engine live out-of-tree as an extension and those who think it's not interesting until it's in core. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Timeline following for logical slots

2016-04-05 Thread Craig Ringer
ints on parent relations, for example. And we're increasingly getting to the point where doing everything all at once is just unmanageable. Still, I don't really want to block work on making logical decoding more real-world usable on inclusion of a logical replication system for PostgreSQL, esp

Re: [HACKERS] Timeline following for logical slots

2016-04-04 Thread Craig Ringer
and integration systems, replicating to non-PostgreSQL databases, etc, if you can't point it at a HA upstream and expect it to still work after failover. Since we can't IMO deliver logical replication based HA in a reasonable timeframe, that means we should really have a way for logical slots to follow physical failover. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Timeline following for logical slots

2016-04-04 Thread Craig Ringer
ready happens more visible. Instead of passing InvalidXLogRecPtr as start_lsn having it transparently replaced with the confirmed_lsn inside CreateDecodingContext, we'd pass the slot->data.confirmed_lsn directly. IMO it'd just make it a little easier to follow what's going on. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Timeline following for logical slots

2016-04-04 Thread Craig Ringer
eful given that it'd really just do a better job of explaining what restart_lsn is and why we start decoding there. So I guess a more detailed comment on the slot struct (not on the field, at the start) would be good. Or that README. Thanks for your patience. -- Craig Ringer http://w

Re: [HACKERS] Timeline following for logical slots

2016-04-04 Thread Craig Ringer
On 4 April 2016 at 14:30, Andres Freund <and...@anarazel.de> wrote: > On 2016-04-04 14:24:52 +0800, Craig Ringer wrote: > > I don't feel like I've grasped this properly yet. I think it's referring > to > > the pg_logical/snapshots/ serialization, the use of which allo

Re: [HACKERS] Timeline following for logical slots

2016-04-04 Thread Craig Ringer
to be crucial for correct function. After all, decoding still restarts at the restart_lsn and feeds relevant xact info into the snapshot builder, accumulates invalidation information, etc. After 9.6 I'd like to go through that, update it, and get it in as a README for logical decoding. It wou

Re: [HACKERS] Timeline following for logical slots

2016-04-04 Thread Craig Ringer
around how logical decoding's parts fit together, and now that there are some other folks in 2ndQ getting involved it seemed like a good idea to start working on making that easier for the next guy. Thanks for linking to your old docs. While a bit outdated as you noted, they'll be simple to update

Re: [HACKERS] Proposal: RETURNING primary_key()

2016-04-03 Thread Craig Ringer
know if that means there are bytes readable from the wrapped SSL socket. The traffic on the underlying socket could be renegotiation messages or whatever. We really need non-blocking reads. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Logical Decoding - Execute join query

2016-04-01 Thread Craig Ringer
results. Depending on what exactly you do. The world doesn't seem to have ended as a result of not immediately dying with an assertion failure in that situation. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] pg_xlogdump fails to handle WAL file with multi-page XLP_FIRST_IS_CONTRECORD data

2016-04-01 Thread Craig Ringer
ach the new testcase once I either get it to reproduce this bug or give up and leave the basic xlogdump testcase alone. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Re: [COMMITTERS] pgsql: Enable logical slots to follow timeline switches

2016-03-31 Thread Craig Ringer
On 1 April 2016 at 12:47, Craig Ringer <cr...@2ndquadrant.com> wrote: > I'll prep a follow-up patch. > > Done and attached. Note that I can't use PG_GETARG_TRANSACTIONID directly since it's a macro defined only in xid.c . It didn't seem worth extracting it and moving it to po

Re: [HACKERS] Timeline following for logical slots

2016-03-31 Thread Craig Ringer
nd as it gets more and more use. I'm trying to help with that, and while I suffer from not knowing it as well as you, that also means I can still see some of the things that might be confusing to newer readers. Then get them wrong when explaining them ;) Speaking of which, did you see the proposed R

Re: [HACKERS] Re: [COMMITTERS] pgsql: Enable logical slots to follow timeline switches

2016-03-31 Thread Craig Ringer
id.c. > Yes, it should. I'll prep a follow-up patch. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] So, can we stop supporting Windows native now?

2016-03-31 Thread Craig Ringer
for 5+ years. That's the main reason I haven't tackled it - so far I've got by pretty well with VS project files + property sheets, and any PGXS-like capability almost certainly couldn't be backported so it's nearly useless for about 5 years. -- Craig Ringer http://www.2ndQuad

Re: [HACKERS] So, can we stop supporting Windows native now?

2016-03-31 Thread Craig Ringer
On 31 March 2016 at 16:20, Andres Freund <and...@anarazel.de> wrote: > On 2016-03-31 09:04:35 +0800, Craig Ringer wrote: > > The cost is small. > > First off I agree we don't want to drop proper windows support. > > But I think "the cost is small" is a pretty

Re: [HACKERS] raw output from copy

2016-03-31 Thread Craig Ringer
hould to report > unknown format. > > PgJDBC does not use libpq. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] raw output from copy

2016-03-31 Thread Craig Ringer
o > > work. Sure, there can be differently implemented clients, but the COPY > > client side is usually simple - store stream to output. > > My point is precisely that I doubt all clients are that stupid about COPY. > PgJDBC definitely isn't. Any changes really need to be tested a

Re: [HACKERS] pg_xlogdump fails to handle WAL file with multi-page XLP_FIRST_IS_CONTRECORD data

2016-03-30 Thread Craig Ringer
plied patch. It passes make check and make -C src/test/recovery check . Marked ready for committer; I'd like to add a TAP test for it, but it's ready to go without that. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] So, can we stop supporting Windows native now?

2016-03-30 Thread Craig Ringer
the build system. Which isn't beautiful, to be sure, but is a burden carried mainly by those who care about Windows support. If we eventually get a CMake build system conversion that'll mostly go away too. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x

Re: [HACKERS] pg_xlogdump fails to handle WAL file with multi-page XLP_FIRST_IS_CONTRECORD data

2016-03-30 Thread Craig Ringer
ff, pageHeaderSize and targetPagePtr could be declared >> inside directly the new while loop. >> > > Thanks Michael for reviewing the patch. I've fixed these issues and new > version is attached. > > Can you describe the process used to generate the sample WAL segment?

Re: [HACKERS] Very small patch for decode.c

2016-03-30 Thread Craig Ringer
origin_id = XLogRecGetOrigin(buf->record); > + RepOriginId origin_id = XLogRecGetOrigin(buf->record); > int i; > Even though it's trivial, created as https://commitfest.postgresql.org/10/594/ so we don't lose it. -- Craig Ringer

Re: [HACKERS] Timeline following for logical slots

2016-03-30 Thread Craig Ringer
y. Andres, I tried to address your comments as best I could. The main one that I think stayed open was about the loop that finds the last timeline on a segment. If you think that's better done by directly scanning the List* of timeline history entries I'm happy to prep a follow-up. -- Craig

Re: [HACKERS] pglogical_output - a general purpose logical decoding output plugin

2016-03-29 Thread Craig Ringer
On 15 March 2016 at 04:48, Andres Freund <and...@anarazel.de> wrote: > On 2016-01-31 05:09:33 +0800, Craig Ringer wrote: > > On 29 January 2016 at 18:16, Andres Freund <and...@anarazel.de> wrote: > > > > > Hi, > > > > > > so, I'm rev

Re: [HACKERS] 2PC support for pglogical

2016-03-25 Thread Craig Ringer
full-table rewrites (alter table ... add column ... default not null) and things that don't (alter table ... drop column ; alter table ... add column ...; without not null or default). -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] NOT EXIST for PREPARE

2016-03-25 Thread Craig Ringer
uff it won't understand that could land up breaking it. Right now we've really got no way to add additional data to query responses, commandcomplete, etc, because we've got no way to make sure the client can cope with it being there. See also https://wiki.postgresql.org/wiki/Todo#Wire_Pro

Re: [HACKERS] NOT EXIST for PREPARE

2016-03-24 Thread Craig Ringer
y doubt you can change this before we do a protocol version bump. The current protocol is too inflexible and doesn't have any kind of capabilities negotiation. I don't think any of those options can work. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] NOT EXIST for PREPARE

2016-03-23 Thread Craig Ringer
very frustrating issue with client-side upsert retry loop approaches. Less of a concern now that 9.5 has a true upsert, but that's not the only area where the client is expected to try it and handle the error if it fails. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] README for src/backend/replication/logical

2016-03-23 Thread Craig Ringer
On 23 March 2016 at 23:54, Alvaro Herrera <alvhe...@2ndquadrant.com> wrote: > Craig Ringer wrote: > > Hi all > > > > As part of some internal training/discussion I wrote some notes on > logical > > decoding's structure and flow that seemed worth polishin

Re: [HACKERS] NOT EXIST for PREPARE

2016-03-23 Thread Craig Ringer
th endless copies of the same PREPARE statements would be good. BTW, PgJDBC doesn't use SQL-level PREPARE anyway, it does it with named portals at the protocol level. You can't add IF NOT EXISTS there, at least not the same way. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] 2PC support for pglogical

2016-03-22 Thread Craig Ringer
d it to callbacks anyway. So I’ve added that info to > abort records. > I was wondering what that was. Again, seems sensible, though I'm not totally convinced of the naming. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Timeline following for logical slots

2016-03-22 Thread Craig Ringer
On 22 March 2016 at 19:48, Andres Freund <and...@anarazel.de> wrote: > On 2016-03-15 21:04:12 +0800, Craig Ringer wrote: > > Thanks very much for the review. > > Are you planning to update the patch? > > Updated patch attached. It removes the questionable cleanups, f

[HACKERS] README for src/backend/replication/logical

2016-03-22 Thread Craig Ringer
and logical replication. It's not really the right place so I'd be happier pointing to a comment somewhere appropriately placed instead. Same with xlog records. The rest is pretty specific to the area of interest. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development

Re: [HACKERS] Timeline following for logical slots

2016-03-22 Thread Craig Ringer
On 22 March 2016 at 19:48, Andres Freund <and...@anarazel.de> wrote: > On 2016-03-15 21:04:12 +0800, Craig Ringer wrote: > > Thanks very much for the review. > > Are you planning to update the patch? > Yes. I just spoke to Álvaro earlier. I'll pick up his modified ver

Re: [HACKERS] Updated backup APIs for non-exclusive backups

2016-03-22 Thread Craig Ringer
her tools would get upset by the addition of new label file entries. Everything we found used regexps to find what they wanted and didn't care in the slightest about new lines. So I wouldn't worry too much. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Developmen

Re: [HACKERS] Applying logical replication changes by more than one process

2016-03-22 Thread Craig Ringer
imaster this assumption is true - even Oracle RAC rarely has two-digit > number of nodes. > But if we want to perform sharding and use logical replication for > providing redundancy, then number of nodes and slots can be essentially > larger. > Sounds like premature optimisation. D

[HACKERS] Re: [HACKERS] Re: [HACKERS] Re: [HACKERS] Re: [HACKERS] Windows service is not starting so there’s message in log: FATAL: "could not create shared memory segment “Global/PostgreSQL.851401618

2016-03-21 Thread Craig Ringer
u may have to dig through a *lot* of output to find it. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Proposal: Generic WAL logical messages

2016-03-21 Thread Craig Ringer
On 18 March 2016 at 20:36, Artur Zakirov <a.zaki...@postgrespro.ru> wrote: > On 17.03.2016 15:42, Craig Ringer wrote: > >> >> >> Would you mind sharing the plugin here? I could add it to >> src/test/modules and add some t/ tests so it runs under the TAP test

Re: [HACKERS] TAP / recovery-test fs-level backups, psql enhancements etc

2016-03-19 Thread Craig Ringer
On 10 March 2016 at 09:53, Craig Ringer <cr...@2ndquadrant.com> wrote: > > Thanks for pushing. > I found a minor issue with the new psql method while writing tests for failover slots. Patch attached. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL De

Re: [HACKERS] postgresql 9.4 on AIX 7.1

2016-03-19 Thread Craig Ringer
ldfarm members, testing with both xlc and gcc builds. http://buildfarm.postgresql.org/cgi-bin/show_members.pl If you're looking for help with that then I suggest posting to pgsql-general rather than -hackers. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Logical decoding slots can go backwards when used from SQL, docs are wrong

2016-03-18 Thread Craig Ringer
The first patch was incorrectly created on top of failover slots not HEAD. Attached patch applies on HEAD. From 87d839f8a2e78abb17fa985502fd5b66f0872b57 Mon Sep 17 00:00:00 2001 From: Craig Ringer <cr...@2ndquadrant.com> Date: Wed, 16 Mar 2016 15:45:16 +0800 Subject: [PATCH 1/2] Correct inc

Re: [HACKERS] Proposal: Generic WAL logical messages

2016-03-18 Thread Craig Ringer
lugin, > which writes DDL queries to the WAL using a hook and replicates this > queries at subscribers. > Would you mind sharing the plugin here? I could add it to src/test/modules and add some t/ tests so it runs under the TAP test framework. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] WIP: Failover Slots

2016-03-16 Thread Craig Ringer
On 15 March 2016 at 21:40, Craig Ringer <cr...@2ndquadrant.com> wrote: > Here's a new failover slots rev, addressing the issues Oleksii Kliukin > raised and adding a bunch of TAP tests. > Ahem, just found an issue here. I'll need to send another revision. --

Re: [HACKERS] Logical decoding slots can go backwards when used from SQL, docs are wrong

2016-03-16 Thread Craig Ringer
patch to amend the docs to reflect the fact that a slot can actually replay the same change more than once. I avoided the strong temptation to update other parts of the docs nearby. Both these are fixes that should IMO be applied to 9.6. -- Craig Ringer http://www

Re: [HACKERS] Soliciting Feedback on Improving Server-Side Programming Documentation

2016-03-15 Thread Craig Ringer
cause I'm still going through the learning curve, I'm probably the least > qualified to write the actual documentation. > You're *extremely* qualified to make notes of what's hard, though, which is something people who've worked on the codebase for a while tend to forget. I've been

Re: [HACKERS] WIP: Failover Slots

2016-03-15 Thread Craig Ringer
logically grouped and clearer. I expect it'd be mostly or entirely squashed for commit. From 256d43f4c8195c893efeb0319d7642853d15f3a9 Mon Sep 17 00:00:00 2001 From: Craig Ringer <cr...@2ndquadrant.com> Date: Tue, 23 Feb 2016 15:59:37 +0800 Subject: [PATCH 1/7] Allow replication slots to

Re: [HACKERS] Timeline following for logical slots

2016-03-15 Thread Craig Ringer
um activity to WAL, and if we replay that WAL *before* we see the slot advance and sync it to the replica, the replica's slot will have an incorrect catalog_xmin that does not reflect the actual on-disk state. Not great. However, if the client is keeping track of its confirmed_lsn (as it must, for correctness) then we know it'll never ask to replay anything older than what it already sent confirmation to the old master for, before failover. Since that's how the slot got advanced and got a new catalog_xmin. That means we won't be attempting to decode anything in the range where the recorded catalog_xmin on the promoted standby after failover would be a problem. The slot will advance to a sensible position when the client specifies the new start lsn. At least, that's my reading of things, and that's what my tests have shown so far. We do start decoding from restart_lsn, so we'll be decoding WAL in the range where catalog_xmin is lying about what's really in the heap, but I don't see anywhere where we're looking at it. It's just collecting up transaction information at that point, right? This same issue will occur if we attempt to do failover slots v2 for 9.7 using non-WAL transport to allow decoding from a replica with failover to cascading standbys, as you mentioned wanting earlier. We'd have to have a way to make sure the slot state on the replica was updated before we replayed past the point in WAL where that slot was updated to the same state on the master. To that end I've thought about proposing a hook to let plugins intercept slot write-out. That way they can take note of the current server LSN and slot state, then make sure they sync that over to the replica before it replays WAL past that LSN. I was really hoping to get failover slots in place so this wouldn't be necessary but it's not looking too promising, and this would provide a somewhat safer way to capture slot advances than just peeking at pg_replication_slots but without having to get the full failover slots stuff in. Having this would at least eliminate the possibility of catalog_xmin being wrong on the replica. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] propose: detail binding error log

2016-03-15 Thread Craig Ringer
gJDBC. Really, the problem is your code: you should setBool not setInteger here. That's really all there is too it. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] propose: detail binding error log

2016-03-14 Thread Craig Ringer
ger); ^ HINT: You will need to rewrite or cast the expression. At the time PostgreSQL parses the statement it doesn't know the parameter values yet, because PgJDBC hasn't sent them to it. It cannot log them even if they mattered, which they don't. -- Cra

Re: [HACKERS] [PATCH] Logical decoding support for sequence advances

2016-03-14 Thread Craig Ringer
On 11 March 2016 at 22:24, Petr Jelinek <p...@2ndquadrant.com> wrote: > On 02/03/16 08:05, Craig Ringer wrote: > >> On 1 March 2016 at 05:30, Petr Jelinek <p...@2ndquadrant.com >> <mailto:p...@2ndquadrant.com>> wrote: >> > > >> >>

Re: [HACKERS] pglogical_output - a general purpose logical decoding output plugin

2016-03-14 Thread Craig Ringer
On 15 March 2016 at 04:48, Andres Freund <and...@anarazel.de> wrote: > On 2016-01-31 05:09:33 +0800, Craig Ringer wrote: > > On 29 January 2016 at 18:16, Andres Freund <and...@anarazel.de> wrote: > > > > > Hi, > > > > > > so, I'm rev

Re: [HACKERS] Timeline following for logical slots

2016-03-14 Thread Craig Ringer
On 15 March 2016 at 07:10, Alvaro Herrera <alvhe...@2ndquadrant.com> wrote: > Petr Jelinek wrote: > > On 04/03/16 17:08, Craig Ringer wrote: > > >I'd really appreciate some review of the logic there by people who know > > >timelines well and preferably know the

Re: [HACKERS] Logical decoding slots can go backwards when used from SQL, docs are wrong

2016-03-14 Thread Craig Ringer
ver lsn you expect to start at, but it's ugly. Ugh. I didn't realise pg_logical_slot_get_changes could backtrack by setting an upto_lsn in the past. That doesn't seem like something we should really be doing - if it's a limit, and we're already past that limit, we should just be returning the empt

Re: [HACKERS] Logical decoding slots can go backwards when used from SQL, docs are wrong

2016-03-14 Thread Craig Ringer
On 11 March 2016 at 20:15, Alvaro Herrera <alvhe...@2ndquadrant.com> wrote: > Craig Ringer wrote: > > Hi all > > > > I think I found a couple of logical decoding issues while writing tests > for > > failover slots. > > > > Despite the docs' cla

Re: [HACKERS] Logical decoding slots can go backwards when used from SQL, docs are wrong

2016-03-11 Thread Craig Ringer
eek. I think the docs for pg_replication_slots are wrong about the confirm location btw, it says data from before the confirm location can't be replayed, but it can over the walsender protocol. There's just no way to ask for it over the SQL interface. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Proposal: RETURNING primary_key()

2016-03-11 Thread Craig Ringer
mes of the primary keys => ORMs > should use "return specific column names" API, not just "return something > generated". > Yep. There are many "should"s. I absolutely agree that this is one of them. One reason some clients do it this way is that earli

Re: [HACKERS] The plan for FDW-based sharding

2016-03-11 Thread Craig Ringer
t usually just bug reports or complaints that whatever you built isn't good enough to meet some random person's particular use case. Ah well. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

[HACKERS] Logical decoding slots can go backwards when used from SQL, docs are wrong

2016-03-11 Thread Craig Ringer
xtensions. Opinions? -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Proposal: RETURNING primary_key()

2016-03-10 Thread Craig Ringer
basic constraint that we should return a set of generated keys in the case where a statement does an insert that adds rows to a table with a SERIAL (or an owned SEQUENCE). Seems like we could do pretty much whatever we want for multiple-generated-columns cases etc. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] TAP / recovery-test fs-level backups, psql enhancements etc

2016-03-09 Thread Craig Ringer
On 10 March 2016 at 05:30, Alvaro Herrera <alvhe...@2ndquadrant.com> wrote: > Craig Ringer wrote: > > > 004 allows filtering on RecursiveCopy by a predicate function. Needed for > > filesystem level backups (007). It could probably be squashed with 007 if > > desired

Re: [HACKERS] Proposal: RETURNING primary_key()

2016-03-09 Thread Craig Ringer
On 10 March 2016 at 00:41, Igal @ Lucee.org <i...@lucee.org> wrote: > On 3/8/2016 5:12 PM, Craig Ringer wrote: > >> One of the worst problems (IMO) is in the driver architecture its self. >> It attempts to prevent blocking by guestimating the server's send buffer >

Re: [HACKERS] Proposal: RETURNING primary_key()

2016-03-09 Thread Craig Ringer
On 9 March 2016 at 21:30, Robert Haas <robertmh...@gmail.com> wrote: > On Tue, Mar 8, 2016 at 8:12 PM, Craig Ringer <cr...@2ndquadrant.com> > wrote: > > There certainly are server/protocol frustrations. > > I'm sympathetic to all of these and think we should work

Re: [HACKERS] the include-timestamp data returned by pg_logical_slot_peek_changes() is always 2000-01-01 in 9.5.1

2016-03-09 Thread Craig Ringer
he second of the millennium...) -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] fun with "Ready for Committer" patches

2016-03-08 Thread Craig Ringer
recisely is required to move a patch from needs-review to ready-for-committer. I've done a chunk of review for a number of patches, but I'm not always confident saying "all clear, proceed". -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Proposal: RETURNING primary_key()

2016-03-08 Thread Craig Ringer
. This would make working with big binary objects in the DB considerably more practical. I'm sure there are others I haven't remembered or run into in there too. See https://wiki.postgresql.org/wiki/Todo#Wire_Protocol_Changes for some. -- Craig Ringer http://www.2ndQuadrant.c

Re: [HACKERS] Proposal: RETURNING primary_key()

2016-03-08 Thread Craig Ringer
e. I realise that's not practical for all apps, which is why supporting the int flag form better is desirable, and we just have to figure out what exactly we should be returning... -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Parallel query fails on standby server

2016-03-08 Thread Craig Ringer
Looks like it might be a good idea to add some tests to src/test/recovery for parallel query on standby servers... -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] How can we expand PostgreSQL ecosystem?

2016-03-07 Thread Craig Ringer
On 7 March 2016 at 20:55, MauMau <maumau...@gmail.com> wrote: > From: Craig Ringer > -- > We could help ORMs solve the N+1 SELECTs problem and help them avoid > transferring vast join projections unnecessarily. That'd make P

Re: [HACKERS] Proposal: RETURNING primary_key()

2016-03-07 Thread Craig Ringer
eed to satisfy the JDBC driver's requirements and anything else that might be related. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] The plan for FDW-based sharding

2016-03-07 Thread Craig Ringer
On 7 March 2016 at 23:02, Robert Haas <robertmh...@gmail.com> wrote: > On Fri, Mar 4, 2016 at 11:17 PM, Craig Ringer <cr...@2ndquadrant.com> > wrote: > > If FDW-based sharding works, I'm happy enough, I have no horse in this > race. > > If it doesn't work I

Re: [HACKERS] How can we expand PostgreSQL ecosystem?

2016-03-07 Thread Craig Ringer
tems I tend to favour doing it right myself. Which is why some of the compromises currently necessary in the logical replication space on Pg irk me so -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] How can we expand PostgreSQL ecosystem?

2016-03-07 Thread Craig Ringer
n appalling security history. Plugins can just talk straight to the DB, and are written by people who have never heard of parametrized queries or, half the time, transactions. What makes it popular and successful is also what makes supporting Pg in a way that'll actually see useful adoption hard. --

Re: [HACKERS] The plan for FDW-based sharding

2016-03-07 Thread Craig Ringer
ing of > SERIALIZABLE transactions remain in doubt, but that would involve > allowing bigger deviations from commit order in transaction > application, which may confuse people. The argument on the other > side is that if they use transaction isolation less strict than > SERIALIZABLE that they ar

Re: [HACKERS] How can we expand PostgreSQL ecosystem?

2016-03-05 Thread Craig Ringer
tell me which you'd look over first if you were evaluating things. > How can we attract software vendors to support PostgreSQL? What words are > convincing to appeal the increasing potential of PostgreSQL as a good > replacement for commercial databases? > Change the name :p -

Re: [HACKERS] Proposal: SET ROLE hook

2016-03-05 Thread Craig Ringer
an do in terms of access to syscache/relcache/heap etc? Why do you pass GetSessionUserId() to the hook given that the caller can look it up directly? Just a convenience? -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] The plan for FDW-based sharding

2016-03-04 Thread Craig Ringer
I want to prevent. Sharding that only-just-works and then stops us getting anything better into core. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] The plan for FDW-based sharding

2016-03-04 Thread Craig Ringer
d be prepared to #if PG_VERSION_NUM around them if you use them. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] The plan for FDW-based sharding

2016-03-04 Thread Craig Ringer
dea how to even begin to tackle it. What you've mentioned here is helpful and I'd be interested if you could share a bit more of your experience in the area. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] The plan for FDW-based sharding

2016-03-04 Thread Craig Ringer
. I don't have a strong position on whether a DTM should be in core or not as I haven't done enough work in the area. I do think it's interesting to strongly require that a DTM be in core while we also reject things like pooling that are needed by a large proportion of users. -- Craig Ri

Re: [HACKERS] The plan for FDW-based sharding

2016-03-04 Thread Craig Ringer
uld have wider applications. It's not unusual for me to be working on something and think "gee, I wish there was a hook here". -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] VS 2015 support in src/tools/msvc

2016-03-04 Thread Craig Ringer
be part of a source tarball so as > it would be possible to compile the code on Windows using MSVC without > having to install MSYS. I haven't done that in this patch, thoughts on > the matter are welcome. > That's consistent with how we include the generated scanner and lexer file

Re: [HACKERS] Equivalent of --enable-tap-tests in MSVC scripts

2016-03-04 Thread Craig Ringer
On 5 March 2016 at 00:10, Alvaro Herrera <alvhe...@2ndquadrant.com> wrote: > Craig Ringer wrote: > > > If it's the result of perltidy changing its mind about the formatting as > a > > result of this change I guess we have to eyeroll and live with it. > perl

Re: [HACKERS] WIP: Failover Slots

2016-03-04 Thread Craig Ringer
On 24 February 2016 at 18:02, Craig Ringer <cr...@2ndquadrant.com> wrote: > I really want to focus on the first patch, timeline following for logical > slots. That part is much less invasive and is useful stand-alone. I'll move > it to a separate CF entry and post it to a separ

Re: [HACKERS] Timeline following for logical slots

2016-03-04 Thread Craig Ringer
On 1 March 2016 at 21:00, Craig Ringer <cr...@2ndquadrant.com> wrote: > Hi all > > Per discussion on the failover slots thread ( > https://commitfest.postgresql.org/9/488/) I'm splitting timeline > following for logical slots into its own separate patch. > > I've u

Re: [HACKERS] Equivalent of --enable-tap-tests in MSVC scripts

2016-03-04 Thread Craig Ringer
on for asserts should not be indented, not sure why it > was, attached is an updated patch). > > If it's the result of perltidy changing its mind about the formatting as a result of this change I guess we have to eyeroll and live with it. perltidy leaves the file alone as it is in th

Re: [HACKERS] TAP / recovery-test fs-level backups, psql enhancements etc

2016-03-04 Thread Craig Ringer
On 4 March 2016 at 20:35, Craig Ringer <cr...@2ndquadrant.com> wrote: > Fix attached. > > Apparently I need a "remind me if you see the word attach in an email" plugin. Sigh. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Developmen

Re: [HACKERS] TAP / recovery-test fs-level backups, psql enhancements etc

2016-03-04 Thread Craig Ringer
eat with a filesystem-level copy, but that there are WAL retention issues (unsurprisingly) when the backup is created without slots then the slot state is synced over by an external client. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

<    5   6   7   8   9   10   11   12   13   14   >